1398304 - Intermittent leakcheck | gpu process: 96 bytes leaked (TextureHost, TextureSourceProvider, nsTArray_base)

Status: RESOLVED FIXED

(Core :: Graphics: Layers, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: wkocher [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=129494840&repo=autoland

https://archive.mozilla.org/pub/firefox/tinderbox-builds/autoland-win64-debug/autoland_win10_64-debug_test-mochitest-clipboard-e10s-bm111-tests1-windows-build5.txt.gz

Comment 1

[Intermittent Failures Robot]

3 failures in 924 pushes (0.003 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 2
* autoland: 1

Platform breakdown:
* windows10-64: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1398304&startday=2017-09-04&endday=2017-09-10&tree=all

Comment 2

[Ryan VanderMeulen [:RyanVM]]

This and bug 1376658 are happening a lot on my 57-as-Beta Try simulations on Win8. Any ideas who might be able to investigate, Milan?

Comment 3

[Nicolas Silva [:nical]]

Hard to tell. I haven't been involved in any recent change that might have affected this. I'd probably look at whether there have been happenings in the GPU process video stuff but that's a wild guess. TextureSourceProvider appears to be related to advanced layers (not sure if anything has changed recently in that area either).
There are too many things that can hold on to (something that can hold on to) a TextureHost for me to think of anything else, but maybe Bas or dvander will have have ideas about the TextureSourceProvider stuff?

Comment 4

[Bas Schouten (:bas.schouten)]

David or Jean-Yves, maybe one of you guys has an idea of something that could have changed in this area?

Comment 5

[Ryan VanderMeulen [:RyanVM]]

*Very* rough regression range (will work on trying to narrow it down a bit):
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=fd87bb184e29&tochange=3a6d2e30bbe1

Comment 6

[Milan Sreckovic [:milan] (needinfo for best results)]

That would suggest advanced layers.

Comment 7

[Milan Sreckovic [:milan] (needinfo for best results)]

I'll leave NI on Bas & David if this is AL related.

Comment 8

[Jean-Yves Avenard [:jya]]

Couldn't this be related to the new try machines using windows 10 finally?

and as such it may have existed before but not be exposed?

Comment 9

[Ryan VanderMeulen [:RyanVM]]

They're happening on Win8 too, so no.

Comment 10

[Ryan VanderMeulen [:RyanVM]]

What about bug 1398659? Regression range is down to https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=fd87bb184e29&tochange=1f1893590a1d and I should have it down to a specific merge in the next 30-45min.

Comment 11

[Jean-Yves Avenard [:jya]]

very unlikely to be bug 1398659, it doesn't touch anything related to IPC

Comment 12

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=bf3d75d48631&tochange=a5f163da8a9b was indeed the regressing merge. So yeah, looks like Advanced Layers.

Comment 13

[Bas Schouten (:bas.schouten)]

I agree this is most likely to be AdvancedLayers, what is holding onto the TextureHost when there's nothing -else- leaking though, that's rather mysterious.. TextureSourceProvider has two arrays so those are the arrays we're leaking. I wonder if it can happen if we end a frame at a certain stage that we have a reference cycle between a TextureHost holding on to a TextureSourceProvider and also being in one of the arrays.

Comment 14

[Intermittent Failures Robot]

21 failures in 247 pushes (0.085 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* try: 18
* mozilla-inbound: 2
* autoland: 1

Platform breakdown:
* windows8-64: 14
* windows10-64: 7

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1398304&startday=2017-09-13&endday=2017-09-13&tree=all

Comment 15

[Ryan VanderMeulen [:RyanVM]]

Bug 1398993 made this go from ~75% failure rate to nonexistent on Beta57 (confirmed via bisection). Too early to say for sure if it had the same effect on regular trunk or not.

Comment 16

[Intermittent Failures Robot]

16 failures in 191 pushes (0.084 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* try: 12
* autoland: 3
* mozilla-central: 1

Platform breakdown:
* windows8-64: 8
* windows10-64: 8

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1398304&startday=2017-09-14&endday=2017-09-14&tree=all

Comment 17

[Cameron McCormack (:heycam)]

FWIW despite bug 1398993's title, it ended up disabling Stylo in any process other than a non-e10s main process or an e10s content process.

Comment 18

[Ryan VanderMeulen [:RyanVM]]

OrangeFactor is still showing some instances of this leak on trunk, but very low-volume.
https://treeherder.mozilla.org/logviewer.html#?job_id=131275907&repo=autoland

Comment 19

[Intermittent Failures Robot]

62 failures in 1032 pushes (0.06 failures/push) were associated with this bug in the last 7 days. 

This is the #45 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* try: 37
* autoland: 12
* mozilla-inbound: 10
* mozilla-central: 3

Platform breakdown:
* windows10-64: 36
* windows8-64: 26

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1398304&startday=2017-09-11&endday=2017-09-17&tree=all

Comment 20

[Intermittent Failures Robot]

18 failures in 190 pushes (0.095 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* autoland: 12
* mozilla-central: 5
* mozilla-inbound: 1

Platform breakdown:
* windows10-64: 18

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1398304&startday=2017-09-20&endday=2017-09-20&tree=all

Comment 21

[Joel Maher ( :jmaher ) (UTC -8)]

I see 50 failures in the last week:
https://brasstacks.mozilla.com/orangefactor/index.html?display=Bug&bugid=1398304

this seems to be across all tests we run on windows10 hardware (newer tests added in the last week primarily).  We don't have a gpu instance for VMs to run the tests- we are working on moving browser-chrome to a VM which would reduce this frequency a bit.  

:ryanvm is there a patch we can land to make this better- I see a lot of back and forth in the previous comments.

Comment 22

[Ryan VanderMeulen [:RyanVM]]

I don't know anything more about this beyond what was said in comment 15.

Comment 23

[Joel Maher ( :jmaher ) (UTC -8)]

:bas.schouten, is there work we could do here to fix this based on your comment 13?

Comment 24

[David Anderson [:dvander] - inactive, e-mail if emergency]

I'll do some diagnostics on try to see if we can't narrow this down.

Comment 25

[Intermittent Failures Robot]

24 failures in 199 pushes (0.121 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* autoland: 9
* mozilla-inbound: 8
* mozilla-central: 5
* try: 2

Platform breakdown:
* windows10-64: 24

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1398304&startday=2017-09-21&endday=2017-09-21&tree=all

Comment 26

[David Anderson [:dvander] - inactive, e-mail if emergency]

Attachment: fix

The only thing I can think of is that the cycle is formed by UnlockAfterComposition being called after layers has been shutdown, which is theoretically possible. If this happens we can just immediately invoke ReadUnlockTextures.

I did three try runs with this patch and didn't see any leaks. A single run without the patch did have leaks.

Comment 27

[Bas Schouten (:bas.schouten)]

Comment on attachment 8911041 [details] [diff] [review]
fix

Review of attachment 8911041 [details] [diff] [review]:
-----------------------------------------------------------------

Good find.

Comment 28

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/4cda16f44d0f
Fix shutdown leak in TextureSourceProviderMLGPU. (bug 1398304, r=bas)

Comment 29

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/4cda16f44d0f

Comment 30

[Ryan VanderMeulen [:RyanVM]]

Please request Beta approval on this when you get a chance.

Comment 31

[Intermittent Failures Robot]

70 failures in 943 pushes (0.074 failures/push) were associated with this bug in the last 7 days. 

This is the #25 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 33
* mozilla-central: 17
* mozilla-inbound: 16
* try: 2
* oak: 2

Platform breakdown:
* windows10-64: 70

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1398304&startday=2017-09-18&endday=2017-09-24&tree=all

Comment 32

[David Anderson [:dvander] - inactive, e-mail if emergency]

Comment on attachment 8911041 [details] [diff] [review]
fix

Approval Request Comment
[Feature/Bug causing the regression]: Advanced Layers
[User impact if declined]: Tiny memory leak on shutdown
[Is this code covered by automated tests?]: Yes
[Has the fix been verified in Nightly?]: Yes
[Needs manual test from QE? If yes, steps to reproduce]:  No
[List of other uplifts needed for the feature/fix]:
[Is the change risky?]: No
[Why is the change risky/not risky?]: Small change to break a reference cycle
[String changes made/needed]:

Comment 33

[Sylvestre Ledru [:Sylvestre]]

Fix a memory leak + intermittent, taking it.
should be in 57b4

Comment 34

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/4280e8315271