1295226 - talos tsvgx regressions when changing mozharness unzip logic

Status: RESOLVED WONTFIX

(Testing :: Talos, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

we have a consistent regression with bug 1258539 where when it landed the first time tsvgx for win7/win8 regressed for e10s.  It was backed out for other failures, we saw the improvements.  A week later it landed again with no talos specifc changes involved, guess what, the same regressions.

here is alerts:
https://treeherder.mozilla.org/perf.html#/alerts?id=2394

win7:
https://treeherder.mozilla.org/perf.html#/comparesubtest?originalProject=autoland&originalRevision=4b946dd315c8cb1c53bb6d666adb2ccaa88d8177&newProject=autoland&newRevision=b9c9444d6a05371ac480f75a9151db00ed296b7b&originalSignature=36c80a0ebefd651b67e61e6634ae20d9c034f2c7&newSignature=36c80a0ebefd651b67e61e6634ae20d9c034f2c7&framework=1

win8:
https://treeherder.mozilla.org/perf.html#/comparesubtest?originalProject=autoland&originalRevision=4b946dd315c8cb1c53bb6d666adb2ccaa88d8177&newProject=autoland&newRevision=b9c9444d6a05371ac480f75a9151db00ed296b7b&originalSignature=79acd47ffc9e36cd6aea558b46ed6667986e21f2&newSignature=79acd47ffc9e36cd6aea558b46ed6667986e21f2&framework=1


or to avoid the links-
win7:
composite-scale-rotate.svg    ~20%
composite-scale-opacity.svg   ~20%
composite-scale-rotate-opacity.svg  0%
rest 0%

win8:
composite-scale-rotate.svg    ~12%
composite-scale-opacity.svg   ~15%
composite-scale-rotate-opacity.svg  15%
rest 0%


so why do we get regressions on these specific files?

here is one of them:
https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/svgx/composite-scale-rotate-opacity.svg

we have a fairly small file, and then this references an image file multiple times.  In fact the tsvg-opacity suite has more svg files which are larger, yet they don't regress?

What I find in common is that we have a file referenced with xlink and we do a transform on it (rotate/scale).  Is it possible that doing this is causing Firefox to do some heavier profile or source file access than we should be doing?  Why would this only affect e10s, and why would it change when we have a regressions?

I have done 3 try pushes with sps profiling to see if that helps us:
https://treeherder.mozilla.org/#/jobs?repo=try&author=[e-mail]&fromchange=56eafbb22e16cd4c850d4ca77c96848c11ea1165&tochange=2e2be89419bd753a87806674faad20006aabbb60

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

:heycam, do you know about svg?  if not- maybe you know who might?  jwatt is afk for a couple months.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

I had forgotten to add the fix for sps profiling, here is what should be the real set of try pushes:
https://treeherder.mozilla.org/#/jobs?repo=try&author=[e-mail]&fromchange=c61b986d9bf8d9e725ae2deff7835e221fefb004&tochange=911ff3a30ddd2085fb75254854c5422bef6035c5

Comment 3

[Cameron McCormack (:heycam)]

> Is it possible that doing this is causing Firefox to do some heavier profile or source
> file access than we should be doing?

The test harness unzipping the test contents in a different way causing performance regressions on these specific tests sounds baffling.  Not impossible, since it's going to poke at the filesystem differently, but I see no good reason why Firefox should be doing something different here with this specific content, at least on the SVG layer.

> Why would this only affect e10s, and why would it change when we have a regressions?

No idea.  Could it be something to do with sandboxing?


Are these tests aimed at measuring rendering performance?  If so, can we make them not reliant on the time taken to load the external images?  Or do you consider the external image loading an important part of the test too?

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

Thanks for the reply :heycam.  I am not too familiar with the intention of these tests, my understanding is that these are testing the performance different SVG rendering techniques.  I am not sure if it is common to load a file via SVG, I would assume so.

What sounds interesting is the sandboxing, how could we test this is affecting things?  Is there preference to change?  This only seems to affect win7 and win8, not winxp (we don't really do e10s on xp), osx, or linux.


on another note, my try pushes to get spsprofiling have failed, I will look into that.

Comment 5

[Cameron McCormack (:heycam)]

(In reply to Joel Maher ( :jmaher) from comment #4)
> Thanks for the reply :heycam.  I am not too familiar with the intention of
> these tests, my understanding is that these are testing the performance
> different SVG rendering techniques.  I am not sure if it is common to load a
> file via SVG, I would assume so.

Loading an external image is moderately common, yes.

> What sounds interesting is the sandboxing, how could we test this is
> affecting things?  Is there preference to change?  This only seems to affect
> win7 and win8, not winxp (we don't really do e10s on xp), osx, or linux.

You could try changing security.sandbox.content.level to 1:

https://dxr.mozilla.org/mozilla-central/rev/054d4856cea6150a6638e5daf7913713281af97d/browser/app/profile/firefox.js#947-957

Comment 6

[Aki Sasaki (not active)]

Complete guess: if python is keeping the zipfile contents in memory after unzipping, that could cause a performance regression.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Aki, this regression is based on teh move from external unzip/tar commands to ZipFile and TarFile. The in-memory download and extraction was done by Armen later on.

Comment 8

[Avi Halachmi (:avih)]

(In reply to Cameron McCormack (:heycam) (away Sep 28) from comment #3)
> Are these tests aimed at measuring rendering performance?

They are.

> If so, can we
> make them not reliant on the time taken to load the external images?

They're not reliant on image loading performance. The tsvgx test measures SVG animation after the page has loaded, and the SVGs are part of the page itself, i.e. not external anything.

> Or do
> you consider the external image loading an important part of the test too?

Not for tsvgx. Other tests which test page load time (tp5o) do depend, and actually measure loading speed, and it does have more/other factors which affect performance, like network access performance (even though the web server from which the content loads is on localhost).

Comment 9

[Cameron McCormack (:heycam)]

OK.  So that test could be changed to avoid using external images, but then we'll need to adjust our talos results expectations, and if we're going to do that, we may as well just accept the regression amount that the unzip patch causes.

Comment 10

[Avi Halachmi (:avih)]

(In reply to Cameron McCormack (:heycam) (away Sep 28) from comment #9)
> OK.  So that test could be changed to avoid using external images ...

In tsvgx the results do not depend on image loading performance... The test itself starts after the load event. See comment 8.