1772368 - Investigate how mitmproxy changes impact regressions

Status: NEW

(Testing :: Raptor, task, P2)

Description

[Greg Mierzwinski [:sparky]]

This bug is for looking into how mitmproxy changes influence the regressions that we capture. See bug 1769903 for the issue which prompted the creation of this bug.

In that bug, we found that once we backed out the patch from here (bug 1755491), there was no more regression even though our autoland graphs very clearly showed the regression from bug 1769903.

It's unclear why this happened, and it raises a few concerns and questions:

1. How does mitmproxy, or our setup of mitmproxy, interfere with Firefox?
2. Do we have other regressions which only occur because of mitmproxy?
3. Are we missing regressions because of our setup in mitmproxy?
4. Should we modify how we treat mitmproxy-based metric changes?

It's well known that mitmproxy changes can change our baseline metrics. What is not well known is how mitmproxy changes influence our ability to see or miss regressions because of some form of interference and this is what we need to investigate in this bug.

Comment 1

[Alexandru Ionescu (needinfo me) [:alexandrui]]

After we had an MVP of mitm7 (lead by kimberly) there were some bugs filed for follow-up on it. Here is the planning doc. Probably those questions would be relevant there.

Comment 2

[Kash Shampur [:kshampur] ⌚EST]

:sparky any recent thoughts/what are some action items for revisiting this bug

given the recent mitm8 upgrade as well as the recent push for re evaluating how we measure noise

Comment 3

[Greg Mierzwinski [:sparky]]

Thanks for checking :kshampur, but I don't think there's much we can do here at the moment. It could be interesting to see if the regression which prompted this bug still happens when we update the code and use mitm8 instead of 7, but I think this is a low priority at the moment.