Closed Bug 1315041 Opened 3 years ago Closed 3 years ago

Split perfherder build_metrics out into more fine-grained buckets

Categories

(Firefox Build System :: General, defect)

Product:
Firefox Build System
Component:
General
Type:
defect
Priority:
Not set

Tracking

(firefox52 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla52
Tracking Flags:
Tracking Status
firefox52 --- fixed

People

(Reporter: ted, Assigned: gps)

References

Details

Attachments

(2 files)

Bug 1315041 - Record environment state in perfherder options;
58 bytes, text/x-review-board-request
wlach
: review+
nthomas
: review+
Details
Bug 1315041 - Revive recording of build phases to perfherder;
58 bytes, text/x-review-board-request
catlee
: review+
Details 
Currently we publish build_metrics to perfherder, which is great, but the results can be hard to interpret. I think we should split them out a bit more. At the very least I'd like to see:
* buildbot vs. taskcluster as separate buckets
* EC2 instance types

We might also want to track clobber vs. dep builds, if possible.

wlach suggests that we simply need to add something like `extraOptions: ['buildbot', 'c4.4xlarge']` to the perfherder data. I believe we'd be putting that here:
https://dxr.mozilla.org/mozilla-central/rev/ade8d4a63e57560410de106450f37b50ed71cca5/testing/mozharness/mozharness/base/python.py#935
Assignee: nobody → gps
Status: NEW → ASSIGNED
Comment on attachment 8807335 [details]
Bug 1315041 - Record environment state in perfherder options;

https://reviewboard.mozilla.org/r/90516/#review90402

::: testing/mozharness/mozharness/base/python.py:59
(Diff revision 3)
> +        opts.append('taskcluster')
> +        # Include the instance type so results can be grouped.
> +        opts.append(os.environ['TASKCLUSTER_INSTANCE_TYPE'])

Alternate idea: we combine the execution environment into 1 field. e.g. `taskcluster-c4.4xlarge`. Not sure what's best for perfherder. That's why wlach gets the review :)
Comment on attachment 8807335 [details]
Bug 1315041 - Record environment state in perfherder options;

This looks good to me (https://treeherder.mozilla.org/perf.html#/graphs?series=%5Btry,df168c600d043d8663ca109d5234e58eb7da74e2,1,2%5D). I'm fine with however you want to format the extra options.
Attachment #8807335 - Flags: review?(wlachance) → review+
FYI nthomas said the instance metadata is available at /etc/instance_metadata.json for buildbot EC2 builders, but he wasn't sure about on Windows (and I have no idea how to figure that out). It's also available over HTTP, but that might be a pain to do reliably.
The only references to "instance_metadata" I see in all my source checkouts are in the build/puppet repo. I'm going to call scope bloat and defer buildbot instance type population to a follow-up bug.

I'll submit a tweaked patch to extra options formatting to make things more forward compatible.
I take that back. I found the code that writes instance_metadata.json and am going to try to make it work. I suspect I won't get it to work on first try. But at least the code will be there to iterate on.
Comment on attachment 8807335 [details]
Bug 1315041 - Record environment state in perfherder options;

https://reviewboard.mozilla.org/r/90516/#review90522

lgtm
Attachment #8807335 - Flags: review?(wlachance) → review+
Comment on attachment 8807758 [details]
Bug 1315041 - Revive recording of build phases to perfherder;

https://reviewboard.mozilla.org/r/90802/#review90538
Attachment #8807758 - Flags: review?(catlee) → review+
Comment on attachment 8807335 [details]
Bug 1315041 - Record environment state in perfherder options;

https://reviewboard.mozilla.org/r/90516/#review90556

Holy cow the try push properly recorded the EC2 instance type for buildbot! Landing this.
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/de69c6acf5c5
Revive recording of build phases to perfherder; r=catlee
https://hg.mozilla.org/integration/autoland/rev/950d65906200
Record environment state in perfherder options; r=wlach
Blocks: 1315410
https://hg.mozilla.org/mozilla-central/rev/de69c6acf5c5
https://hg.mozilla.org/mozilla-central/rev/950d65906200
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
status-firefox52: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla52
Comment on attachment 8807335 [details]
Bug 1315041 - Record environment state in perfherder options;

https://reviewboard.mozilla.org/r/90516/#review90642

On Windows you can use C:\etc\instance_metadata.json. We're only using c4.2xlarge right now but may add a range in bug 1315158.
Attachment #8807335 - Flags: review?(nthomas) → review+
Blocks: 1317999
Blocks: 1318002
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.