1608241 - Raptor tests pass and report metric data, even with mitmproxy exit errors

Status: RESOLVED DUPLICATE of bug 1587387

(Testing :: Raptor, defect, P1)

Description

[Stephen Donner [:stephend] Not actively reading bugmail]

Raptor tests pass and report metric data, even with mitmproxy exit errors

xref: bug 1607750

STR:

1. Load https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=284103686&repo=try&lineNumber=4340
2. Notice this error: ERROR - raptor-mitmproxy Error: Mitmproxy exited with error code 572
3. Note the missing content in these screenshots: https://firefoxci.taskcluster-artifacts.net/PZAEzFt0RNeH1XNAMMESLA/0/public/test_info/screenshots.html, and compare with https://firefoxci.taskcluster-artifacts.net/IPeC5zuPQh2H0We1P56mzg/0/public/test_info/screenshots.html
4. Note that Raptor returns code 0: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=284103686&repo=try&lineNumber=4523
5. And thus, the job: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=284103686&repo=try&lineNumber=4544

Actual Results:

Raptor returns code 0 and passes, although it's clear from the screenshots that the full content didn't load & render, and mitmproxy exited with error code 572.

There's also a case where screenshots are fine: https://firefoxci.taskcluster-artifacts.net/IPeC5zuPQh2H0We1P56mzg/0/public/test_info/screenshots.html,
but we still have the mitmproxy error: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=284107929&repo=try&lineNumber=4153

In either case, I don't think we can trust the performance metrics we're submitting to Perfherder.

Expected Results:

I think Raptor should fail the test if, somehow, we can detect mitmproxy errors via mozproxy; relevant mitm.py (mozproxy) code, here: https://searchfox.org/mozilla-central/rev/ba4fab1cc2f1c9c4e07cdb71542b8d441707c577/testing/mozbase/mozproxy/mozproxy/backends/mitm/mitm.py#270-281

Comment 1

[Stephen Donner [:stephend] Not actively reading bugmail]

Some updates:

1. the error code 572 is propagated from Windows (https://docs.microsoft.com/en-us/windows/win32/debug/system-error-codes--500-999-) and is analogous to CTRL-C'ing the app, so it might be a bit of a red herring, but we should be sure.
2. there's already also bug 1587387 (which I read, but forgot to mention, here) which :bebe is already looking into

Comment 2

[Florin Strugariu [:Bebe]]

(In reply to Stephen Donner [:stephend] from comment #1)

Some updates:

1. the error code 572 is propagated from Windows (https://docs.microsoft.com/en-us/windows/win32/debug/system-error-codes--500-999-) and is analogous to CTRL-C'ing the app, so it might be a bit of a red herring, but we should be sure.

we stop mitm using .kill()
https://searchfox.org/mozilla-central/source/testing/mozbase/mozproxy/mozproxy/backends/mitm/mitm.py#270
which simulates a CTR+C (I think)

2. there's already also bug 1587387 (which I read, but forgot to mention, here) which :bebe is already looking into

I'm investigating an implementation of a retry process to start mitm

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Can you provide an update on your investigation?

Comment 4

[Florin Strugariu [:Bebe]]

after some investoation I found out that error code 572 is expected in windows cases.
ProcessHandler uses winprocess.ERROR_CONTROL_C_EXIT to stop windows processes
https://searchfox.org/mozilla-central/source/testing/mozbase/mozprocess/mozprocess/processhandler.py#152

because of this we removed the error message for mitm when we stop the process:
https://phabricator.services.mozilla.com/D59743

I would mark this as won't fix as mitm 572 error code on windows is not an issue.

See also: bug 1587387

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Thanks Bebe