1579844 - 4.57 - 12.85% Heap Unclassified / Images (windows10-64, windows10-64-qr, windows10-64-shippable-qr) regression on push cb1dbe32e1556ac3dc205803e429c9e3ea3304e9 (Wed September 4 2019)

Status: RESOLVED WONTFIX

(Testing :: General, defect)

Description

[Alexandru Ionescu (needinfo me) [:alexandrui]]

We have detected an awsy regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=cb1dbe32e1556ac3dc205803e429c9e3ea3304e9

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

13% Images windows10-64-shippable-qr opt 7,336,633.48 -> 8,279,661.23
13% Images windows10-64-shippable-qr opt 7,349,520.31 -> 8,287,997.16
12% Images windows10-64-qr opt 7,407,256.60 -> 8,279,434.55
8% Images windows10-64 opt 5,582,309.75 -> 6,010,815.84
7% Heap Unclassified windows10-64-shippable-qr opt 54,492,337.68 -> 58,112,616.64
5% Heap Unclassified windows10-64-qr opt 55,041,580.31 -> 58,016,214.41
5% Heap Unclassified windows10-64 opt 48,727,095.69 -> 50,955,827.79

Improvements:

4% Resident Memory windows10-64 opt 611,901,021.56 -> 585,380,001.74
3% Resident Memory windows10-64-qr opt stylo tp6 722,335,054.80 -> 703,724,316.57

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=23002

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/AWSY/Tests

Comment 1

[Andrew Halberstadt [:ahal]]

I believe these regressions were expected as we purposefully downgraded to a lower end machine (for comparatively large cost savings). That said these numbers are all Greek to me and I have no idea if the scope of these regressions are representative of what was expected.

Jeff, can you confirm whether this looks ok or redirect to someone who can?

Thanks!

Comment 2

[Bobby Holley (:bholley)]

Hm, that's a bit odd. How different are the GPUs and drivers in the machines we switched across?

Comment 3

[Andrew Halberstadt [:ahal]]

I believe we switched from g3.4xlarge to g3s.xlarge. Here's a comparison chart I found (which is from a blog post so may be outdated):
https://aws.amazon.com/about-aws/whats-new/2018/10/introducing-a-new-size-for-amazon-ec2-g3-graphics-accelerated-instances/

Beyond that I have no idea, maybe Jeff would be able to add a bit more.

Edit: Same chart here: https://aws.amazon.com/ec2/instance-types/g3, and both instances have GPUs with 8GB of memory

Comment 4

[Jeff Muizelaar [:jrmuizel]]

It seems like this could be caused by GPU driver, available RAM etc. Since there was no code change I don't think there's anything that we need to worry about here. A particular baseline is basically arbitrary. I say WONTFIX.

Comment 5

[Bobby Holley (:bholley)]

Can you verify from the logs that we're at least getting the same graphics backend / acceleration for each platform before and after the change?

Comment 6

[Jeff Muizelaar [:jrmuizel]]

Unfortunately, I don't think we dump that information into the logs. I filed bug 1580244 about trying to get it.