1804300 - Move mitmproxy crash logging to a separate file

Status: RESOLVED FIXED

(Testing :: Raptor, defect, P1)

Description

[Kash Shampur [:kshampur] ⌚EST]

https://bugzilla.mozilla.org/show_bug.cgi?id=1804281#c0

tests are passing and videos look correct, however looking at the logs it seems mitmproxy is crashing and re-recording the sites does not seem to help.

immediate options I can think of:

• Reverting these tests to temporarily replay on mitm 6/7 should remove these crashlogs while we investigate. The advantage here is we already know mitm 6/7 worked here before
• I could even introduce mitm9 binaries which seem to not have these long crash logs at all, and only restrict these 4/5 tests to replay on 9 (9.0.1 as of writing this). The benefit here is we don't need to bring back a duplicate of the old alt-serverplayback.py, which would be required for using mitm 7. And by restricting it this way, there's only going to be a slight disturbance in baselines for a few tests versus everything, when doing mitm 8 upgrades (either way, reverting this to mitm 7 will also disrupt the baseline as well)

edit looking over the tests, it seems mitm7 replaying mitm 6 recordings (netflix, instagram, fb-responsiveness) has the same error log. So it seems the temporary quickest solution here is to revert those 3 tests back to mitm 6, and wikia to mitm7.

Comment 1

[Greg Mierzwinski [:sparky]]

:kshampur, what do you think about captuing and/or writing the mitmproxy output to a file? That way we could keep the same version and still have mitmproxy logging.

Comment 2

[Kash Shampur [:kshampur] ⌚EST]

As discussed in triage, mitmproxy moved their logging into their own logging in version 9, so that is why we don't see the message in the regular log file but do see it in the mitmproxy.log e.g. https://firefoxci.taskcluster-artifacts.net/LheMhuOZR2qUzLbTXGtN1w/0/public/test_info/mitmproxy.log
So, the crash is still there.

Also, in favor of not going backwards, we won't reverse the mitmproxy version

So what we can do for now is write logic to capture these mitm specific error logs and output them to another file.

:sparky since mitm9 already outputs it to it's own mitmproxy.log, what do you think about using that for these specific tests?
Or would it be better to leave the binary version as is for now, and control the logging output ourself (and I guess it would probably easier to add a warning that way)

Comment 3

[Greg Mierzwinski [:sparky]]

Let's control the logging ourselves because it should be pretty easy to make this change. See here for where we start the recording: https://searchfox.org/mozilla-central/source/testing/mozbase/mozproxy/mozproxy/backends/mitm/mitm.py#376-377

Could you file a follow-up bug to look into mitm9 and get a jira ticket for it?

Comment 4

[Kash Shampur [:kshampur] ⌚EST]

Yep sounds good!

Comment 5

[Kash Shampur [:kshampur] ⌚EST]

Attachment: Bug 1804300 - Move mitmproxy error logging to a separate file. r?#perftest

Comment 6

[Kash Shampur [:kshampur] ⌚EST]

Attachment: Bug 1804300 - Move mitmproxy error output to existing mitmproxy log. r?#perftest

This patch moves mitmproxy specific error logging into the existing mitmproxy.log file that we use.
This should help in dealing with the 100 lines of failure limit on Treeherder e.g. bug 1804281

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0ae846fcd9c8
Move mitmproxy error output to existing mitmproxy log. r=perftest-reviewers,sparky

Comment 8

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/0ae846fcd9c8