791695 - memory leak in jenkins CI

Status: RESOLVED INVALID

(Firefox :: Untriaged, defect)

Description

[Mihai Sucan [:msucan]]

User Agent: Opera/9.80 (X11; Linux x86_64; U; fr) Presto/2.10.289 Version/12.02

Steps to reproduce:

I use the Jenkins CI web app for a few hours, in about 6-7 tabs. See http://jenkins-ci.org/


Actual results:

Firefox is very slow. It freezes for a few seconds, every few seconds. Firefox becomes unusable. After I close it, it continues to use all CPU resources. I have to kill Firefox.

Comment 1

[Mihai Sucan [:msucan]]

about:memory showed one compartment using over 1gb of memory for the jenkins web app page. A second compartment of the same web app showed about 300 mb mem usage. All of this ... long after closing all of the tabs of the web app.

about:addons on the system where this problem occurs shows: flash, acrobat reader, google talk, icedtea, vlc plugin. No extensions installed beyond the default ones on Ubuntu 12.04.

Please let me know how to provide further help on finding the cause of this bug. Thank you!

(reporting this bug on behalf of a friend who has Firefox 15 on Ubuntu 12.04)

Comment 2

[(mostly gone) XtC4UaLL [:xtc4uall]]

See Bug 740162.

Comment 3

[Mihai Sucan [:msucan]]

Can we mark this as fixed/duplicate of some bug now? Thank you!

Comment 4

[Andrew McCreight [:mccr8]]

Bug 826521 may make it easier to figure out what the problem is, but isn't a fix by itself.

This bug does look similar to bug 751557, maybe.

Comment 5

[Liviu Cirdei]

Hi Mihai, 

Can you still reproduce this on one of the latest Firefox versions? E.g. Firefox 43 and Nightly 46?

Comment 6

[Liviu Cirdei]

Closing as incomplete due to lack of response from the reporter. Fell free to reopen the bug if you can reproduce the issue on a current build.

Comment 7

[Viachaslau_Kabak]

Having the same issue
please reopen this bug - very annoying issue

Comment 9

[Viachaslau_Kabak]

Attachment: ff_memory_leak_issue.PNG

FF RAM usage after 11 hours work

Comment 10

[ovidiu boca[:Ovidiu]]

Hi Viachaslau,

Can you please follow the steps from here https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Reporting_a_Performance_Problem   and attach the Gecko profile. Thanks

Comment 11

[Viachaslau_Kabak]

Hi, 
Started reproducing the issue
as soon as it is reproduced - I will attache the Gecko profile
thanks

Comment 12

[Viachaslau_Kabak]

https://perfht.ml/2udLUCD

Comment 13

[Viachaslau_Kabak]

Sorry for a long delay - several times lost a moment to get a profile - FF memory was over 15 Gb and it was freezed

Comment 14

[ovidiu boca[:Ovidiu]]

Mike can you please take a look at this Gecko profile? Thanks

Comment 15

[Mike Conley (:mconley) (:⚙️)]

Unfortunately, the profile in comment 12 isn't super useful because it was captured on a non-Nightly build of Firefox. I can tell from it, however, that we're doing a lot of cycle collection in the parent process.

Are you able to reproduce this bug with all of your add-ons disabled?

Comment 16

[ovidiu boca[:Ovidiu]]

Viachaslau_Kabak can you please download FF Nightly from here: https://nightly.mozilla.org/ and then follow again the steps from comment 10 and send the new profile capture. Also are you able to reproduce this bug with all of your add-ons disabled?

Comment 17

[Viachaslau_Kabak]

Hi, in progress
will update as soon as results are ready

Comment 18

[Viachaslau_Kabak]

Hi,
have a results according running FF in safe mode (w/o addons)
FF using 3-5 Gb RAM but often freezing (seems to flush unused memory)
During viewing pages with pipelines FF memory usage was increased to 15Gb but after closing pages it returned to 4Gb without FF restart
Do I still need to use nightly release and get gecko profile?

Comment 19

[Viachaslau_Kabak]

Attachment: FF addons installed

Comment 20

[ovidiu boca[:Ovidiu]]

Thanks, Viachaslau_Kabak

We need to wait until Mike can look over your gecko profile, but it's no need to continue using Nightly if you don't want.

Mike, please see if the capture helps you. Thanks

Comment 21

[Viachaslau_Kabak]

Ok, but I installed nightly to check)
Found that it splits the FF to several processes - so need to check all processes memory usage together

Comment 22

[Viachaslau_Kabak]

This is a gecko profile from my VM (i have installed nightly build there)
https://perfht.ml/2vdD7Ow

Comment 23

[Viachaslau_Kabak]

https://perfht.ml/2veeEZg

Comment 24

[Viachaslau_Kabak]

this a new profile

Comment 25

[Mike Conley (:mconley) (:⚙️)]

If this is a leak, I don't think the Gecko Profiler is going to give us anything except confirmation. It won't tell us what is leaking or where, really. We'd need an about:memory report to have a better shot at knowing what the Jenkins web app is leaking.

There's also the Memory Snapshot DevTool, which might be particularly useful since we think we know which site is leaking: https://developer.mozilla.org/en-US/docs/Tools/Memory/Basic_operations#Taking_a_heap_snapshot

The profile posted by Viachaslau_Kabak shows something frequently painting - there's probably some kind of animation going on, but it appears to be happening at least somewhat smoothly.

I also see some JS causing sync layout flushes, but the biggest problem definitely appears to be a long GC at the end: https://perf-html.io/public/7916bdeaf03d7986d90589d98cd4d6e37274cbf4/calltree/?hiddenThreads=6-5-4-7-1&range=598.1861_646.2581&search=GarbageCollect&thread=8&threadOrder=0-2-4-5-6-7-8-1-3

We need to figure out what this web app is leaking.

Comment 26

[Viachaslau_Kabak]

Mike, thanks a lot
I will try to get memory report

Comment 27

[Viachaslau_Kabak]

Hi, 
attached the FF memory snapshot and memory report

Comment 28

[Viachaslau_Kabak]

Attachment: Memory report

Comment 29

[Viachaslau_Kabak]

Memory snapshot is about 60Mb size (maximum allowed is 10Mb)
where i can store the file?

Comment 30

[Viachaslau_Kabak]

Hi all,
any new about this issue?

Comment 31

[ovidiu boca[:Ovidiu]]

Hi Mike, can you please take a look at Viachaslau memory report? When you have time, thanks.

Comment 32

[Mike Conley (:mconley) (:⚙️)]

A few things stand out:

1) There are more than 4 content processes in this report, which is a non-default configuration, and it makes me wonder how else the reporter has customized their browser. Getting a copy of about:support might be useful.

2) There's a lot of heap unclassified in the main process, which is very interesting. I have no idea where that's coming from, or what to do about that. :/

Hey erahm, did I miss anything from that reading?

Comment 33

[Eric Rahm [:erahm]]

(In reply to Mike Conley (:mconley) from comment #32)
> A few things stand out:
> 
> 1) There are more than 4 content processes in this report, which is a
> non-default configuration, and it makes me wonder how else the reporter has
> customized their browser. Getting a copy of about:support might be useful.

Sure their settings may make things worse, but they reproduced in safe-mode too.

> 2) There's a lot of heap unclassified in the main process, which is very
> interesting. I have no idea where that's coming from, or what to do about
> that. :/
> 
> Hey erahm, did I miss anything from that reading?

This seems about right.

Background for those who don't want to go down a 5 year old rabbit hole: Previously we declared this to be a bug in Jenkins (see bug 751557), marked it for evangelism, got no response from the devs and gave up.

The high amount of heap-unclassified is concerning so I'd like to at least address that, the best way to figure out what's going on is to use DMD. I'm not sure we can do much about Jenkins generating a ton of orphan nodes other than to report the issue again.

So my questions are:
1) Can you reproduce in another browser? Chrome or Edge would be a good test case.
2) Can you provide steps to reproduce? Specifically I need a link to a Jenkins instance
3) Can you provide a memory report with safe-mode enabled?

Comment 34

[Viachaslau_Kabak]

Hi,
about:support info is attached
according to Eric's questions:
1) I will try in chrome
2) I can help you with testing pipeline script to run it on installed jenkins (mine is not exposed to outside)
3) will do it

Comment 35

[Viachaslau_Kabak]

Attachment: aboutSupport_raw.txt

Comment 36

[Viachaslau_Kabak]

Attachment: aboutSupport_text.txt

Comment 37

[Viachaslau_Kabak]

Attachment: memory-report_insafemode.json.gz

Comment 38

[Viachaslau_Kabak]

p. 3 is done - memory report is attached

Comment 39

[Viachaslau_Kabak]

Attachment: 14453373_snapshot_in_safe mode.zip.001

Comment 40

[Viachaslau_Kabak]

Attachment: 14453373_snapshot_in_safe mode.zip.002

Comment 41

[Viachaslau_Kabak]

Attachment: 14453373_snapshot_in_safe mode.zip.005

Comment 42

[Viachaslau_Kabak]

Attachment: 14453373_snapshot_in_safe mode.zip.003

Comment 43

[Viachaslau_Kabak]

Attachment: 14453373_snapshot_in_safe mode.zip.004

Comment 44

[Viachaslau_Kabak]

attached memory dump from dev tools

Comment 45

[Viachaslau_Kabak]

Hi Eric,
According testing in Chrome - the same issue
Seems that the problem in jenkins pipelines...

Later I will send you a script and instructions

thanks

Comment 46

[ovidiu boca[:Ovidiu]]

Viachaslau_Kabak,

This is a bug in Jenkins like Eric suggested in comment 33?

Comment 47

[Viachaslau_Kabak]

Hi,
possible this is a bug in jenkins cause the chrome has leak too
could you please help to open this bug in jenkins bug report system with description and put the link here
I will vote and try to help jenkins team will all needed info

thanks a lot!

Comment 48

[ovidiu boca[:Ovidiu]]

Hi,

Unfortunately, I can't help you with this because I have a lot of work on Photon. Eric do you have some time and help Viachaslau_Kabak with this? You also have more experience with what's going on here. Thanks

Comment 49

[Eric Rahm [:erahm]]

(In reply to Viachaslau_Kabak from comment #47)
> Hi,
> possible this is a bug in jenkins cause the chrome has leak too
> could you please help to open this bug in jenkins bug report system with
> description and put the link here
> I will vote and try to help jenkins team will all needed info
> 
> thanks a lot!

I discussed this briefly on IRC with Jenkins devs, they pointed me to an existing Jenkins bug: https://issues.jenkins-ci.org/browse/JENKINS-45094. I've added a comment linking to this bug, but please feel free to add details there!

Comment 50

[Viachaslau_Kabak]

Eric, thanks a lot
I have commented Jenkins bug and will help them with info

Comment 51

[ovidiu boca[:Ovidiu]]

Thanks Eric for your help. Based on the fact that this is not a Firefox issue, I will mark it as invalid.