1425787 - [meta] Binary implementation of mozsystemmonitor for Linux/Win/Mac

Status: NEW

(Firefox Build System :: General, enhancement)

Description

[Gregory Szorc [:gps]]

mozsystemmonitor is a Python package providing a mechanism to record system resource usage.

We've historically had difficulties getting that working on as many platforms as we want because of Python C extension issues. Plus, the code is somewhat cobbled together and is a bit brittle.

In support of bug 1425785, I'd like to create a Rust implementation of this functionality.

While we want to eventually support all tier 1 platforms, our immediate goal is to integrate this into run-task. And run-task runs mainly on Linux. So as our initial goal, let's get the Rust implementation working on Linux.

Comment 1

[Gregory Szorc [:gps]]

Attachment: Bug 1425787 - Rust library for collecting system resource metrics

This commit adds a Rust crate providing an implementation of a system
resource collector.

The crate provides functionality similar to the mozsystemmonitor Python
package. Unlike the Python version (which uses psutil), the Rust version
has no runtime dependencies outside of the system. This makes it easy
to drop into a statically linked binary so it can be used more
universally.

While we have an eventual goal for this crate to run on all tier 1
supported platforms, our immediate goal is to get it deployed on
Linux. So that is where the bulk of the code lives.

Review commit: https://reviewboard.mozilla.org/r/208190/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/208190/

Comment 2

[Gregory Szorc [:gps]]

I'm not actively working on this.

Comment 3

[Simon Fraser [:sfraser] ⌚️GMT]

I'll take this over, unless any objections are raised.

Comment 4

[Simon Fraser [:sfraser] ⌚️GMT]

The main goal as I understand it is to ease deployment, and lower any performance hit when taking the measurements. run-task itself could have this sort of feature added but it doesn't currently have any external dependency, and adding one would be a pain, especially something with compiled components like psutil.

Current platform breakdown for tasks which use run-task on mozilla-central is
linux 5185
win 3402
gecko-t-bitbar-gw-perf-g5 522
gecko-t-bitbar-gw-perf-p2 776
osx 838
images 76
gecko-t-bitbar-gw-unit-p2 12

So while Linux is still the priority, Windows and OSX support are blockers.

There's https://github.com/rust-psutil/rust-psutil which seems to not yet be feature complete. For comparison there's also https://github.com/shirou/gopsutil since the deployment strategy there would be the same, and the worker is already Go.

psutil	rust-psutil			gopsutil
Feature	Linux	Windows	MacOSX	Linux	Windows	MacOSX
System
cpu_times	✅	❌	✅	✅	✅	✅
cpu_percent	✅	❌	✅	✅	✅	✅
virtual_memory	✅	❌	✅	✅	✅	✅
swap_memory	✅	❌	✅	✅	✅	✅
net_io_counters	✅	❌	✅	✅	✅	✅
disk_io_counters	✅	❌	❌	✅	✅	✅
Process
parent	✅	❌	❌	✅	✅	✅
children	❌	❌	❌	✅	✅	✅
cpu_times	✅	❌	✅	✅	✅	✅
cpu_percent	✅	❌	✅	✅	✅	✅
memory_info	✅	❌	✅	✅	✅	✅
io_counters	❌	❌	❌	✅	✅	✅

Comment 5

[Simon Fraser [:sfraser] ⌚️GMT]

We had a discussion about this in Taskcluster RFC 160, as one of the options open to us is to include this as a feature inside the worker.

The summary of that discussion is that resource monitoring as a worker feature has advantages, such as being payload and deployment agnostic, allowing use across multiple clusters. However, it would be safer to develop this as an in-tree feature and have it controlled by the payload, at least for the moment. We will be iterating on code, output format, processing pipeline, reports and so on; once all those aspects have stabilised we can revisit closer worker integration.

Taking the current Mixin as v1, this v2 approach will be to create a toolchain containing the task monitor, add it to task definitions using existing tool dependency mechanisms, and control its deployment using task definition changes. Later, v3 may be improvements on this or inclusion into the workers themselves.

Regardless of other details this would mean the current recording of phases will be lost, so we should discuss a means of recording those and combining them later in the processing pipeline for reports.

Comment 6

[BugBot [:suhaib / :marco/ :calixte]]

The bug assignee didn't login in Bugzilla in the last 7 months.
:mhentges, could you have a look please?
For more information, please visit auto_nag documentation.