1550235 - Add performance dashboards for AWSY tests

Status: RESOLVED FIXED

(Tree Management Graveyard :: Web Tools, task, P2)

Description

[Eric Rahm [:erahm]]

We'd like to add a performance dashboard for the Are We Slim Yet data. Currently we can perform queries in perfherder, but it would be nice to have a dedicated site for viewing predefined queries. arewefastyet.com is a good example of what we're looking for.

In order of preference, the end product should:

1. Resurrect areweslimyet.com - we are on friendly terms with the owner
2. Stand up a new site that is explicitly mozilla owned
3. Just tack the data on to arewefastyet.com

As far as UI goes just a total copy of arewefastyet.com is acceptable.

Relevant data

Framwork

All of the data we care about is currently under the awsy framework, this includes results from the following tests:

• awsy - full load of tp5
• awsy tp6 - full load of tp6
• awsy lite - awsy on android
• awsy base - content process overhead

Platforms

We care about shippable configurations from the mozilla-central project. For windows we want to focus on the qr platform.

Standard AWSY data

The first two (awsy and the tp6 variant) report the following test suites:

• Resident Memory
• Explicit Memory
• Heap Unclassified
• JS
• Images

Notes: awsy lite only reports the Resident Memory suite. awsy tp6 uses a "tp6" extraOption to differentiate itself.

Each of those suites has 9 subtests:

• Fresh start
• Fresh start [+30s]
• After tabs open
• After tabs open [+30s]
• After tabs open [+30s, forced GC]
• Tabs closed extra processes
• Tabs closed
• Tabs closed [+30s]
• Tabs closed [+30s, forced GC]

AWSY base

This test reports slightly different information. It reports the following suites:

• Base Content Resident Unique Memory
• Base Content Heap Unclassified
• Base Content JS
• Base Content Explicit

Each of those suites has 1 subtest:

• After tabs open [+30s, forced GC]

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The priority flag is not set for this bug.
:bc, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 2

[Bob Clary [:bc] (inactive)]

erahm: I'm setting this to P5. Feel free to bump it higher.

armenzg: Is this something you would be interested in?

Comment 3

[Armen [:armenzg]]

I will be discussing this with kthiessen. Maybe Q3.

Comment 4

[Eric Rahm [:erahm]]

(In reply to Bob Clary [:bc:] from comment #2)

erahm: I'm setting this to P5. Feel free to bump it higher.

armenzg: Is this something you would be interested in?

This is a P1 at this point, we need to have a dashboard that's visible to people outside of the memshrink project.

Comment 5

[Eric Rahm [:erahm]]

Karl, is this still on your team's radar?

Comment 6

[Karl Thiessen [:kthiessen, he/him]]

As far as I know, Armen is still aware of it. Armen, do you have time to work on this in Q3?

Comment 7

[Armen [:armenzg]]

This might be doable for Q3. Earliest on August.

Which people can validate the work?

I could provide a similar solution to AWFY with little effort.
Changes beyond that will need further planning and evaluate resource allocation.

Comment 8

[Eric Rahm [:erahm]]

(In reply to Armen [:armenzg] from comment #7)

This might be doable for Q3. Earliest on August.

August would work fine for us.

Which people can validate the work?

Myself, Neha Kochar, and Dave Hunt would all be good people to reach out to.

I could provide a similar solution to AWFY with little effort.
Changes beyond that will need further planning and evaluate resource allocation.

Just having something like AWFY would be great! Once we have something in place we can decide if there are any other changes we'd like in the future.

Comment 9

[Armen [:armenzg]]

I've started poking at this but it is going to slip into September.

Comment 10

[Armen [:armenzg]]

erahm: Could you please have a look at this? https://5d6e6acbb103df000870bb96--firefox-performance-dashboard.netlify.com/win10/overview?numDays=60

I have not been able to find the Android jobs.

We could overlay base and tp6 benchmarks if you wish to.

Comment 11

[Eric Rahm [:erahm]]

(In reply to Armen [:armenzg] from comment #10)

erahm: Could you please have a look at this? https://5d6e6acbb103df000870bb96--firefox-performance-dashboard.netlify.com/win10/overview?numDays=60

This looks great, thank you Armen! A few minor notes:

1. We want to use the shippable variant of each platform, ie linux64-shippable
2. On Windows we want to highlight the qr variant, ie windows10-64-shippbable-qr

I have not been able to find the Android jobs.

Here's an example: https://treeherder.mozilla.org/perf.html#/graphs?timerange=31536000&series=mozilla-central,1707028,1,4

We could overlay base and tp6 benchmarks if you wish to.

I think we want to keep them separate, it's a somewhat different dataset.

Comment 12

[Eric Rahm [:erahm]]

One other small item: the units should be bytes.

Comment 13

[Armen [:armenzg]]

I have done another quick pass (see commit).

It is now using the shippable variant, I have added win10 qr and Android:

I've created a separate Netlify app using the awsy branch:
https://awsy.netlify.com

I still need to tackle the bytes comment, however, is everything else meeting your needs?

Comment 14

[Eric Rahm [:erahm]]

(In reply to Armen [:armenzg] from comment #13)

I still need to tackle the bytes comment, however, is everything else meeting your needs?

It's looking good, thanks for the quick changes.

Comment 15

[Armen [:armenzg]]

Attachment: Showing Bytes

erahm, I fixed the yLabel.

What's left for this to meet your needs?
https://awsy.netlify.com/win10/overview?numDays=60

Comment 16

[Eric Rahm [:erahm]]

(In reply to Armen [:armenzg] from comment #15)

Created attachment 9091430 [details]
Showing Bytes

erahm, I fixed the yLabel.

What's left for this to meet your needs?
https://awsy.netlify.com/win10/overview?numDays=60

That looks great. My only other request (if it's reasonably easy) is that given the size of our values on the x-axis it would be nice to format the numbers so that they include separators, ie: 15,000,000 instead of 15000000.

Comment 17

[Armen [:armenzg]]

Attachment: Add commas to numbers on y ticks

Comment 18

[Armen [:armenzg]]

Hello Eric,
I see you're on leave. I will also be for the next 3 weeks.

If you have any futher requests you can do so directly on https://github.com/mozilla-frontend-infra/firefox-performance-dashboard/issues

If you need someone to work on it before I come back please talk with Karl.

Comment 19

[Armen [:armenzg]]

Hi Eric,
Does the dashboard meet your needs?