Closed Bug 1572356 Opened 8 months ago Closed 4 months ago

Register the vcs framework in perfherder

Categories

(Tree Management :: Perfherder, task, P2)

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: glandium, Assigned: airimovici)

Details

Attachments

(1 file)

I was looking around in build logs and saw that there's data like this in the logs:

PERFHERDER_DATA: {"framework": {"name": "vcs"}, "suites": [{"extraOpt
ions": ["c4.4xlarge"], "lowerIsBetter": true, "name": "clone", "serverUrl": "hg.mozilla.org", "shoul
dAlert": false, "subtests": [], "value": 1.5160000324249268}, {"extraOptions": ["c4.4xlarge"], "lowe
rIsBetter": true, "name": "update", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests":
 [], "value": 584.4290001392365}, {"extraOptions": ["c4.4xlarge"], "lowerIsBetter": true, "name": "o
verall", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 586.776000022
8882}, {"extraOptions": ["c4.4xlarge"], "lowerIsBetter": true, "name": "overall_pull", "serverUrl": 
"hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 586.7760000228882}, {"extraOptions"
: ["c4.4xlarge"], "lowerIsBetter": true, "name": "overall_pull_fullcheckout", "serverUrl": "hg.mozil
la.org", "shouldAlert": false, "subtests": [], "value": 586.7760000228882}, {"extraOptions": ["c4.4x
large"], "lowerIsBetter": true, "name": "overall_pull_emptywdir", "serverUrl": "hg.mozilla.org", "sh
ouldAlert": false, "subtests": [], "value": 586.7760000228882}]}

(from https://queue.taskcluster.net/v1/task/TC9heKv7TLKfmKfxs8Vb_A/runs/1/artifacts/public/logs/live_backing.log)

But I can't find this anywhere in https://treeherder.mozilla.org/perf.html#/graphs

(In reply to Mike Hommey [:glandium] from comment #0)

I was looking around in build logs and saw that there's data like this in the logs:

PERFHERDER_DATA: {"framework": {"name": "vcs"}, "suites": [{"extraOpt
ions": ["c4.4xlarge"], "lowerIsBetter": true, "name": "clone", "serverUrl": "hg.mozilla.org", "shoul
dAlert": false, "subtests": [], "value": 1.5160000324249268}, {"extraOptions": ["c4.4xlarge"], "lowe
rIsBetter": true, "name": "update", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests":
 [], "value": 584.4290001392365}, {"extraOptions": ["c4.4xlarge"], "lowerIsBetter": true, "name": "o
verall", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 586.776000022
8882}, {"extraOptions": ["c4.4xlarge"], "lowerIsBetter": true, "name": "overall_pull", "serverUrl": 
"hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 586.7760000228882}, {"extraOptions"
: ["c4.4xlarge"], "lowerIsBetter": true, "name": "overall_pull_fullcheckout", "serverUrl": "hg.mozil
la.org", "shouldAlert": false, "subtests": [], "value": 586.7760000228882}, {"extraOptions": ["c4.4x
large"], "lowerIsBetter": true, "name": "overall_pull_emptywdir", "serverUrl": "hg.mozilla.org", "sh
ouldAlert": false, "subtests": [], "value": 586.7760000228882}]}

(from https://queue.taskcluster.net/v1/task/TC9heKv7TLKfmKfxs8Vb_A/runs/1/artifacts/public/logs/live_backing.log)

But I can't find this anywhere in https://treeherder.mozilla.org/perf.html#/graphs

I need to look this up. First thing I noticed is vcs isn't among our registered frameworks, which include:

talos
build_metrics
autophone
awsy
awfy
platform_microbench
talos-aws
servo-perf
hasal
raptor
js-bench
devtools
Priority: -- → P2

As I suspected: Perfherder skips data for unregistered frameworks. This is where it does the check.

:glandium if you want to use Perfherder as a visualization tool for this framework, please file a separate bug for registering it.

Status: NEW → RESOLVED
Closed: 8 months ago
Resolution: --- → INVALID
Flags: needinfo?(mh+mozilla)

Let it be that bug, since it has all the context.

Status: RESOLVED → REOPENED
Flags: needinfo?(mh+mozilla)
Resolution: INVALID → ---
Summary: VCS perf data is not visible on perfherder → Register the vcs framework in perfherder

Although... I guess we could swipe that in the build_metrics framework.... It all depends what registering the vcs framework on perfherder does to the data that has been there all along? Would all that data be visible, or would only new data be available?

(In reply to Mike Hommey [:glandium] from comment #5)

Although... I guess we could swipe that in the build_metrics framework.... It all depends what registering the vcs framework on perfherder does to the data that has been there all along? Would all that data be visible, or would only new data be available?

Only new vcs data will be available. Old one has been skipped on PERHERDER_DATA parse time.
It will create a new dropdown item in Perfherder, similar to build_metrics, talos, awsy, raptor...
So, if vcs could me mixed into build_metrics, I suggest go with 1st option.

I found out the data is actually already in ActiveData, so changing the framework is not quite as nice... OTOH... maybe ActiveData is enough...

(In reply to Mike Hommey [:glandium] from comment #7)

I found out the data is actually already in ActiveData, so changing the framework is not quite as nice... OTOH... maybe ActiveData is enough...

If you go with ActiveData, we can close this ticket.

Let's close this for now. I'll reopen if ActiveData ends up not being enough.

Status: REOPENED → RESOLVED
Closed: 8 months ago8 months ago
Resolution: --- → INVALID

Ionuț, would you mind registering the vcs framework? perfherder is a somewhat more straightforward option for visualizing this data. It'll be useful to help us breakdown which pieces of our build are taking the most time and measuring improvements and regressions over time.

Status: RESOLVED → REOPENED
Flags: needinfo?(igoldan)
Resolution: INVALID → ---

(In reply to Eric Rahm [:erahm] from comment #10)

Ionuț, would you mind registering the vcs framework? perfherder is a somewhat more straightforward option for visualizing this data. It'll be useful to help us breakdown which pieces of our build are taking the most time and measuring improvements and regressions over time.

Alex, do you have the cycles to pick this up?

Flags: needinfo?(igoldan) → needinfo?(airimovici)
Type: defect → task

I'll take it

Assignee: nobody → airimovici
Flags: needinfo?(airimovici)
Status: REOPENED → RESOLVED
Closed: 8 months ago4 months ago
Resolution: --- → FIXED

Once the production deploy happens, vcs metrics will be available in Perfherder.

You need to log in before you can comment on or make changes to this bug.