1632639 - [mozproxy] Raptor/AWSY tests can leave mitmdump processes behind with each of those consuming 100% cpu load

Status: RESOLVED FIXED

(Testing :: Raptor, defect, P3)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: Screenshot 2020-04-23 at 22.20.38.png

I noticed again a huge CPU load on my machine and when checking the activity monitor a couple of mitmproxy processes were running with each of them utilizing 100% of CPU load. See screenshot attached.

Most likely this happens when the Raptor job gets aborted eg. Ctrl+C, but maybe also in other situations.

This might have a really bad impact on our CI machines, especially the Mac Minis. We should get this investigated and fixed.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I can also see this for AWSY now. Basically each time you break a running mach raptor/awsy test job with Ctrl+C a mitmdump instance is left behind. Usually it doesn't consume any CPU but can hold a quite amount of memory, e.g one of these instances locally uses 377MB! Then after a while or due to some other reason such processes go crazy and the reported CPU usage can be seen.

Comment 3

[Alex Finder]

This is still a valid issue, most frequently reproducible on Android. Less frequently on Desktop platforms.

Comment 4

[Kash Shampur [:kshampur] ⌚EST]

Attachment: Bug 1632639 - Add signal handler to remove rogue mitmdump processes in AWSY/Raptor. r?#perftest

Comment 5

[Kash Shampur [:kshampur] ⌚EST]

I added a WIP patch that seems to work for me. I think only edge case not accounted for is when Exceptions are hit, otherwise things like keyboard interrupt and session terminations are handled

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/656bae3199dc
https://hg.mozilla.org/integration/autoland/rev/cafcdd4b5b92
Add signal handler to remove rogue mitmdump processes in AWSY/Raptor. r=perftest-reviewers,sparky

Comment 7

[rperta]

https://hg.mozilla.org/mozilla-central/rev/cafcdd4b5b92