1270276 - Record mozharness step data in perfherder

Status: RESOLVED WONTFIX

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Gregory Szorc [:gps]]

I would like to see individual mozharness step data (mainly wall execution times) recorded in Perfherder. This will help us see where time is being spent in mozharness (which is now the bulk of most jobs/tasks in automation).

We currently record this data in influxdb. But that requires a login. Perfherder is in the open and will facilitate more/better data analysis.

Comment 1

[Kyle Lahnakoski [:ekyle]]

You can pull it from ActiveData [1]:

>{
>	"select":[
>		"_id",
>		"etl.timestamp",
>		"build.branch",
>		"action.start_time",
>		"builder.step",
>		"harness.step",
>		"harness.duration"
>	],
>	"from":"jobs.action.timings",
>	"where":{"and":[
>		{"exists":"harness.step"},
>		{"gt":{"action.start_time":"{{today-month}}"}},
>		{"lt":{"action.start_time":"{{today}}"}}
>	]},
>	"limit":1000
>}

Also, there is a build times dashboard [2] that uses this (limited to builds).  It needs work, and could certainly be faster.   The detailed view [3] allows you to click on a data point and show the mozharness steps.  Maybe a selector, that only shows the particular step if interest, is a good feature?

[1] http://activedata.allizom.org/tools/query.html#query_id=O98V+shI
[2] http://people.mozilla.org/~klahnakoski/MoBuildbotTimings/Builds-Overview.html
[3] http://people.mozilla.org/~klahnakoski/MoBuildbotTimings/Builds-Detail.html#sampleMax=2016-05-06&sampleMin=2016-03-01&product=Firefox&type=PGO&platform=Windows64

Comment 2

[Gregory Szorc [:gps]]

Oh, I didn't realize you were ingesting this already! How are you getting the data? Log parsing?

Comment 3

[Kyle Lahnakoski [:ekyle]]

Yes, the text logs are parsed [1]

[1] https://github.com/klahnakoski/TestLog-ETL/blob/fbbd75ed76ff472e1a8a12665dd84b3d3fc011a3/testlog_etl/transforms/pulse_block_to_job_logs.py

Comment 4

[Gregory Szorc [:gps]]

According to wlach in #treeherder, there might be some value in redundantly adding the mozharness step data to perfherder. e.g. we can get alerts when times of certain steps changes significantly.

I guess the only question is whether we should add the data to perfherder via mozharness or whether ekyle's existing data loader could send the data to perfherder.

ekyle: do you have an opinion? (FWIW I don't think it's much effort to record the basic times in mozharness. I already had a patch in progress before you enlightened me about ActiveData.)

Comment 5

[Kyle Lahnakoski [:ekyle]]

If you know what you want to measure, and there are few things to measure, then you may want to use PerfHerder, which will provide you with charts and level-shift detection.  ActiveData is better at a tracking a large number of measures, with many properties, that can be explored and decided at query time, rather than at submission time. 

I do not know if PerfHerder accepts submissions from remote clients, and I doubt it can handle the full job data volume; given the number of steps we are tracking in ActiveData now.

Comment 6

[Kyle Lahnakoski [:ekyle]]

PerfHerder may attempt sending data requests to ActiveData directly, rather than its local database.  This would give it access to more data, without the work of scaling out its database.

Comment 7

[William Lachance (:wlach)]

(In reply to Kyle Lahnakoski [:ekyle] from comment #6)
> PerfHerder may attempt sending data requests to ActiveData directly, rather
> than its local database.  This would give it access to more data, without
> the work of scaling out its database.

Indeed, it might be interesting at some point to get ActiveData to ingest perfherder's database somehow for queries where SQL just isn't enough (I think this is the premise of Periscope: https://www.periscopedata.com/), but that's definitely beyond the scope of this bug. :)

Just adding a PERFHERDER_DATA line to your scripts is definitely the easiest path forward. I can provide advice on formatting that (but just searching for that string in mozilla-central should give you lots of inspiration).

Comment 8

[Armen [:armenzg]]

gps: are you planning on looking into this bug this quarter?

Comment 9

[Gregory Szorc [:gps]]

(Please use needinfo)

No I'm not. I think wlach and ekyle have this under control?

Comment 10

[William Lachance (:wlach)]

I'm not sure if we want to do this: it might be enough just to record the data in treeherder (bug 1258861). Let's leave this open for now just in case.