1272202 - Upload resource monitor JSON as job artifact

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Gregory Szorc [:gps]]

The resource monitor integrated with mozharness has the ability to export its data as JSON. We should upload this JSON file as a job artifact.

Maybe once this is done we could integrate an HTML viewer into Treeherder. We already have an HTML viewer in mozilla-central (`mach resource-usage`). It is very crude and ugly. But it gets the job done.

Comment 1

[Gregory Szorc [:gps]]

Attachment: MozReview Request: Bug 1272202 - Upload raw resource usage data; r?jlund

Data is too valuable to waste. Let's upload the raw resource
data captured by the resource monitor so we can look at resource
usage in more detail whenever we want.

Review commit: https://reviewboard.mozilla.org/r/52237/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/52237/

Comment 2

[Gregory Szorc [:gps]]

Comment on attachment 8751831 [details]
MozReview Request: Bug 1272202 - Upload raw resource usage data; r?jlund

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/52237/diff/1-2/

Comment 3

[Jordan Lund (:jlund)]

Comment on attachment 8751831 [details]
MozReview Request: Bug 1272202 - Upload raw resource usage data; r?jlund

https://reviewboard.mozilla.org/r/52237/#review49187

non issues. just some fyi's about mozharness core offerings

::: testing/mozharness/mozharness/base/python.py:513
(Diff revision 1)
>              self._log_resource_usage()
> +
> +            # Upload a JSON file containing the raw resource data.
> +            try:
> +                upload_dir = self.query_abs_dirs()['abs_upload_dir']
> +                with open(os.path.join(upload_dir, 'resource-usage.json'), 'wb') as fh:

could use self.dump_config[1] here but I think I prefer this as it provides try/catch and uses the more readable with statement

[1] https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/base/script.py#

::: testing/mozharness/mozharness/base/python.py:517
(Diff revision 1)
> +                upload_dir = self.query_abs_dirs()['abs_upload_dir']
> +                with open(os.path.join(upload_dir, 'resource-usage.json'), 'wb') as fh:
> +                    json.dump(self._resource_monitor.as_dict(), fh,
> +                              sort_keys=True, indent=4)
> +            except (AttributeError, KeyError):
> +                self.warning('could not upload resource usage JSON: %s' %

fyi - you can use self.exception(message, level=WARNING)[2] and it will extend the log output with the traceback for your

[2] https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/base/log.py#157

Comment 4

[Gregory Szorc [:gps]]

Comment on attachment 8751831 [details]
MozReview Request: Bug 1272202 - Upload raw resource usage data; r?jlund

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/52237/diff/2-3/

Comment 5

[Gregory Szorc [:gps]]

Comment on attachment 8751831 [details]
MozReview Request: Bug 1272202 - Upload raw resource usage data; r?jlund

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/52237/diff/3-4/

Comment 6

[Jordan Lund (:jlund)]

Comment on attachment 8751831 [details]
MozReview Request: Bug 1272202 - Upload raw resource usage data; r?jlund

https://reviewboard.mozilla.org/r/52237/#review49209

::: testing/mozharness/mozharness/base/python.py:512
(Diff revision 4)
>              self._resource_monitor.stop()
>              self._log_resource_usage()
> +
> +            # Upload a JSON file containing the raw resource data.
> +            try:
> +                upload_dir = self.query_abs_dirs()['abs_blob_upload_dir']

hm, come to think of it. I wonder if every job that will have a abs_blob_upload_dir defined? be interesting to see which jobs raise the exception you have defined.

Comment 7

[Gregory Szorc [:gps]]

https://reviewboard.mozilla.org/r/52237/#review49209

> hm, come to think of it. I wonder if every job that will have a abs_blob_upload_dir defined? be interesting to see which jobs raise the exception you have defined.

The exceptions I've seen so far are due to the resource monitor not having an `as_dict` method. I reckon we aren't running a modern version of that package in automation. Time to sift through virtualenv logs. Yay.

Comment 8

[Gregory Szorc [:gps]]

Attachment: mozsystemmonitor-0.1.tar.gz

Here is a source archive for the mozsystemmonitor Python package. Please upload it to our public PyPI mirror at http://pypi.pub.build.mozilla.org/pub/.

Comment 9

[Gregory Szorc [:gps]]

Attachment: MozReview Request: Bug 1272202 - Require psutil >= 3.1.1 in mozsystemmonitor; r?jlund

We have packages for 3.1.1 uploaded to our PyPI server used
for automation. There have been a number of bug fixes since the
version of psutil currently listed. Let's ensure we're running
a modern psutil to minimize our exposure to bugs on older
versions.

Review commit: https://reviewboard.mozilla.org/r/52325/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/52325/

Comment 10

[Gregory Szorc [:gps]]

Attachment: MozReview Request: Bug 1272202 - Bump version of mozsystemmonitor to 0.1; r?jlund

We're currently running version 0.0 in automation. This version
doesn't have as_dict(), which means we can't easily save data
to JSON.

Bump the version to 0.1 in preparation of releasing a new
version.

Review commit: https://reviewboard.mozilla.org/r/52327/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/52327/

Comment 11

[Gregory Szorc [:gps]]

Attachment: MozReview Request: Bug 1272202 - Bump psutil and mozsystemmonitor package versions; r?jlund

mozsystemmonitor 0.1 has been released. It features support
for retrieving a dict of gathered data, making JSON export easy.
Let's use it. This version also requires psutil>=3.1.1, so bump
that version as well.

Review commit: https://reviewboard.mozilla.org/r/52329/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/52329/

Comment 12

[Gregory Szorc [:gps]]

Comment on attachment 8751831 [details]
MozReview Request: Bug 1272202 - Upload raw resource usage data; r?jlund

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/52237/diff/4-5/

Comment 13

[Jordan Lund (:jlund)]

(In reply to Gregory Szorc [:gps] from comment #8)
> Created attachment 8751962 [details]
> mozsystemmonitor-0.1.tar.gz
> 
> Here is a source archive for the mozsystemmonitor Python package. Please
> upload it to our public PyPI mirror at
> http://pypi.pub.build.mozilla.org/pub/.

done:

curl -I http://pypi.pub.build.mozilla.org/pub/mozsystemmonitor-0.1.tar.gz ...
HTTP/1.1 200 OK

Comment 14

[Jordan Lund (:jlund)]

Comment on attachment 8751964 [details]
MozReview Request: Bug 1272202 - Require psutil >= 3.1.1 in mozsystemmonitor; r?jlund

https://reviewboard.mozilla.org/r/52325/#review49255

Comment 15

[Jordan Lund (:jlund)]

Comment on attachment 8751965 [details]
MozReview Request: Bug 1272202 - Bump version of mozsystemmonitor to 0.1; r?jlund

https://reviewboard.mozilla.org/r/52327/#review49257

Comment 16

[Jordan Lund (:jlund)]

Comment on attachment 8751966 [details]
MozReview Request: Bug 1272202 - Bump psutil and mozsystemmonitor package versions; r?jlund

https://reviewboard.mozilla.org/r/52329/#review49259

Comment 17

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/1dfd7b10f77d
https://hg.mozilla.org/integration/mozilla-inbound/rev/e2f85e14c439
https://hg.mozilla.org/integration/mozilla-inbound/rev/332a89cc267b
https://hg.mozilla.org/integration/mozilla-inbound/rev/3e1834b1d9ff

Comment 18

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/1dfd7b10f77d
https://hg.mozilla.org/mozilla-central/rev/e2f85e14c439
https://hg.mozilla.org/mozilla-central/rev/332a89cc267b
https://hg.mozilla.org/mozilla-central/rev/3e1834b1d9ff