Add new Raptor test job for generating cpu and memory info during idle scenario test
Categories
(Testing :: Raptor, defect, P1)
Tracking
(firefox70 fixed)
| Tracking | Status | |
|---|---|---|
| firefox70 | --- | fixed |
People
(Reporter: rwood, Assigned: krubin, Mentored)
References
(Blocks 1 open bug)
Details
Attachments
(1 file, 4 obsolete files)
Bug 1544470 landed the ability to generate cpu info while running of a Raptor test (at the start). The next step is to create a new taskcluster test job that runs the Raptor scenario idle test [0] but with the --cpu-info and --memory-info flags set. This will start generating some cpu and memory info on android and posting in perfherder.
This new test should only run on mozilla-central and on try server, tier 2 to start.
[0] https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/raptor-scn-power-idle.ini
|
Reporter | |
Comment 1•5 years ago
|
||
:davehunt, :sparky: Do you want this combined with the existing idle power test maybe? i.e. add the '--cpu-test' and '--memory-test' flags to this existing test:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&tier=1%2C2%2C3&searchStr=raptor%2Cidl
Or do you want a separate test job for cpu and memory that doesn't do the power test at the same time?
|
||
Comment 2•5 years ago
|
||
If the cpu and memory test flags don't add additional overhead (from the cpu-test patch it doesn't seem like they do) then I think this would be fine to have. It could let us monitor CPU utilization vs. power usage to see if they correlate well with each other as well. I'm curious to know what :davehunt thinks of this approach though.
|
||
Comment 3•5 years ago
|
||
It makes sense to combine them so long as this doesn't adversely affect the other measurements. I'd suggest pushing a few try jobs with these individually and combined. We can retrigger them a few times and see if there's any impact.
|
|
Reporter | |
Comment 4•5 years ago
|
||
Now that Chris has returned to his team, whom should we assign this issue to? Thanks!
|
|
||
Comment 5•5 years ago
|
||
(In reply to Dave Hunt [:davehunt] [he/him] ⌚️UTC from comment #3)
It makes sense to combine them so long as this doesn't adversely affect the other measurements. I'd suggest pushing a few try jobs with these individually and combined. We can retrigger them a few times and see if there's any impact.
Thinking more on this, we can expect the cpu and memory to differ when we're not plugged into USB, and using WiFi. I think it's probably okay for us to measure cpu and memory whilst also measuring power, but we should probably also have a job that measures them without the power data. Does that make sense? So we'd have idle-cmp (cpu, memory, power), and idle-cm (cpu, memory).
(In reply to Robert Wood [:rwood] from comment #4)
Now that Chris has returned to his team, whom should we assign this issue to? Thanks!
I think this could be a good task for Ken.
|
|
||
Comment 7•5 years ago
|
||
I was just commenting on the patch and then thought I would be better to comment here and make sure we're all on the same page. At this point the patch includes a fair number of additional jobs. Is this for the purposes of validating that the cpu and memory gathering does not have an adverse affect on the results of the speedometer and power tests?
Assuming enabling cpu and memory does not affect the other results, I would expect the following jobs as a result of this bug:
- raptor-speedometer-geckoview-power, Rap(sp-cmp)
- raptor-speedometer-geckoview, Rap(sp-cm)
- raptor-scn-power-idle-fenix, Rap-fenix(idl-cmp)
- raptor-scn-idle-fenix, Rap-fenix(idl-cm)
- raptor-scn-power-idle-bg-fenix, Rap-fenix(idlbg-cmp)
- raptor-scn-idle-bg-fenix, Rap-fenix(idlbg-cm)
- raptor-scn-power-idle-fennec, Rap-fennec(idl-cmp)
- raptor-scn-idle-fennec, Rap-fennec(idl-cm)
- raptor-scn-power-idle-bg-fennec, Rap-fennec(idlbg-cmp)
- raptor-scn-idle-bg-fennec, Rap-fennec(idlbg-cm)
- raptor-scn-power-idle-geckoview, Rap(idl-cmp)
- raptor-scn-idle-geckoview, Rap(idl-cm)
- raptor-scn-power-idle-bg-geckoview, Rap(idlbg-cmp)
- raptor-scn-idle-bg-geckoview, Rap(idlbg-cm)
- raptor-scn-power-idle-refbrow, Rap-refbrow(idl-cmp)
- raptor-scn-idle-refbrow, Rap-refbrow(idl-cm)
- raptor-scn-power-idle-bg-refbrow, Rap-refbrow(idlbg-cmp)
- raptor-scn-idle-bg-refbrow, Rap-refbrow(idlbg-cm)
I'll double check, but I don't think there are any jobs in addition to the ones you list.
Note: It may appear like more new stuff was added because I've changed the power tests to have the --cpu-test and --memory-test flags set, and I've renamed them to be, for example, raptor-speedometer-geckoview-cpu-memory-power, rather than just raptor-speedometer-geckoview-power. Namely, I used the convention to include the specific resources read in the test. If you don't want the more verbose names, I can change "cpu-memory-power" back to "power" (but still include the --cpu-test and --memory-test flags ...)
|
Assignee | |
Comment 10•5 years ago
|
||
|
||
Updated•5 years ago
|
|
|
||
Updated•5 years ago
|
|
|
Assignee | |
Comment 11•5 years ago
|
||
|
|
||
Updated•5 years ago
|
|
|
||
Comment 12•5 years ago
|
||
Try run for the latest patch: https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=c617ef03f565b1705f8076f94dc8cfe3c2f1f598
The failures are legitimate and require further work.
|
|
||
Comment 13•5 years ago
|
||
This patch adds two new test tasks for each of the 4 Android browsers running speedometer, idle, or idle-background. These tests gather (i) cpu+memory usage, or (ii) cpu+memory+power usage information. The power tests are left as they are for now to determine if there is an impact to our measurements if we run all resource-usage tests at once. Furthermore, the cron job for android power tests was also modified to prevent it from picking up these tasks.
The cpu usage functons needed to be fixed here because they were not working on MotoG5/Android7 for two reasons: (i) device version was not being detected properly, and (ii) ADB shell commands can't use the pipe functionality (|). Tests needed to be modified to compensate for these changes as well.
|
|
||
Comment 14•5 years ago
|
||
Try run for the latest patch: https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=04c3a0677f1a1ba16f756660af4676f477667d4f
|
|
||
Updated•5 years ago
|
|
||
Comment 15•5 years ago
|
||
Pushed by gmierz2@outlook.com: https://hg.mozilla.org/integration/autoland/rev/2248396d2a7b Add new raptor tests in CI for generating cpu and memory info. r=perftest-reviewers,rwood
|
||
Comment 16•5 years ago
|
||
| bugherder | ||
Description
•