1547932 - allow the host / port option in raptor manifests

Status: RESOLVED WONTFIX

(Testing :: Raptor, defect, P1)

Description

[Tarek Ziadé (:tarek)]

right now, if we use the "use_live_sites" option, it's required to add the --host option when running raptor, otherwise Raptor will set it to 127.0.0.1 and activate the proxy mode.

We want to:

• allow the host option in ini files
• raise an error in case "use_live_sites" is set but not host (via the ini file or command line)

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

This is specifically needed for stream tests which would have to be run in live mode at least for the beginning.

If you consider that to not be allowed, we will have to figure out something else.

One thing could be that the test url could be actually parsed for the host name and port, which would at least not require an additional test setting.

Comment 2

[Tarek Ziadé (:tarek)]

Inferring it from the test_url is a good idea !

I also don't see any reason not to allow the host option in ini files. We might have cases in the future
were we want a host different from the test url (the proxy is not on the same host, the IP is different from 127.0.0.1 etc)

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

As info for Rob and Dave, the problem is that the dependent perf profile as included here sets up a proxy:

https://searchfox.org/mozilla-central/rev/b756e6d00728dda4121f8278a744381d8643317a/testing/profiles/profiles.json#4
https://searchfox.org/mozilla-central/rev/b756e6d00728dda4121f8278a744381d8643317a/testing/profiles/perf/user.js#78-80

Comment 4

[Robert Wood [:rwood]]

Hey guys, I just tried it again with the latest inbound.

If I set 'use_live_sites = true' in raptor-tp6-1.ini, and run ./mach raptor-test --test raptor-tp6-google-firefox --page-cycles 5 it works great - I can see that the live site is loaded and the measurements taken. Mitmproxy is not started up at all.

Same as on android, with my GP2 device connected to WiFi. If I set 'use_live_sites = true' in raptor-tp6m-3.ini, then run ./mach raptor-test --test raptor-tp6m-google-restaurants-geckoview --app=geckoview --binary=org.mozilla.geckoview_example --page-cycles 5 it also works great - I can see that the live site is loaded (not a recording) and the measurements taken. Mitmproxy is not started up at all.

So 'use_live_sites' is working as/is... maybe I'm not understanding the issue correctly? Does the above not work for you? Thanks!

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The test raptor-tp6-google uses a HTTPS address but not HTTP. As such you don't see the problem.

You could replicate by changing the proxy in perf/ to also be a HTTPS one.

Comment 6

[Robert Wood [:rwood]]

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+1] from comment #5)

The test raptor-tp6-google uses a HTTPS address but not HTTP. As such you don't see the problem.

You could replicate by changing the proxy in perf/ to also be a HTTPS one.

Ahh I see, and most of the Raptor URL's are HTTPS. This probably explains the timeout I was seeing with using ESPN as a live site (Bug 1547491).

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Just remembered that Tarek set up port 8080 for our current mirror:
http://yttest.dev.mozaws.net:8080/2019/main.html?test_type=playbackperf-test&command=run:8

So maybe just switching to that port for now is fine, and we leave the proxy in place with the caveat that live mode doesn't work with sites running on this port.

Note that I'm still trying to figure out why this proxy is set. Lets wait for bug 1546880.

Comment 8

[Tarek Ziadé (:tarek)]

I think the bottom line is that if we're using the live site there's no reason to run a proxy at all, so we need to make sure that this function is called if use_live_site is true:

https://searchfox.org/mozilla-central/source/testing/raptor/raptor/raptor.py#313

or the proxy not set in the first place in user.js

any other fix sounds to me like a workaround - not sure why we would not want to do the right fix here.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The proxy actually prevents Firefox services and tests to reach remote URLs as told by jmaher. Whereby I have to say that the former is mostly done by just modifying the appropriate services preferences, and changing the host to the locally running httpd instance. So a change there might be more appropriate, which might allow us to even disable/remove those proxy settings. Lets follow up on bug 1546880 for the proxy stuff.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+2]]

This feature is actually not needed. It can simply be fixed by removing the proxy settings from the perf profile. I will fix that along with bug 1546880.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Well, actually mark it as wontfix for now. Maybe it might be necessary in the future.