Closed Bug 1546880 Opened 7 months ago Closed 6 months ago

Remove usage of proxy server from testing/profiles/perf/user.js

Categories

(Testing :: General, defect, P1)

Version 3
defect

Tracking

(firefox69 fixed)

RESOLVED FIXED
mozilla69
Tracking Status
firefox69 --- fixed

People

(Reporter: whimboo, Assigned: whimboo)

References

(Blocks 1 open bug)

Details

Attachments

(3 files)

As of right now the Raptor profiles get a proxy server setup for HTTP here:

https://searchfox.org/mozilla-central/source/testing/profiles/perf/user.js#79-81

Whenever I run a test against a live site, the connection cannot be made because it's routed through the proxy which doesn't run at this point.

I don't know for which reasons this proxy exists for perf tests, so maybe Andrew and Joel can give some answers to it.

Also why is the proxy only set for HTTP but not HTTPS too? Most connections nowadays run through a secure connection.

Flags: needinfo?(jmaher)
Flags: needinfo?(ahal)

this is set for all of our testing. It appears there are prefs for ssl:
https://searchfox.org/mozilla-central/source/testing/awsy/conf/tp6-prefs.json#7

IIRC tp6/mitmproxy has to turn that off so we can use the proxy that mitmproxy provides.

Flags: needinfo?(jmaher)

I have no idea :)

Flags: needinfo?(ahal)

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #1)

this is set for all of our testing. It appears there are prefs for ssl:

For which specific reason does the proxy exist for perf tests?

https://searchfox.org/mozilla-central/source/testing/awsy/conf/tp6-prefs.json#7

This is awsy only so doesn't affect Raptor.

IIRC tp6/mitmproxy has to turn that off so we can use the proxy that mitmproxy provides.

Flags: needinfo?(jmaher)
See Also: → 1547932

I gave you an example, talos uses the same prefs as well and so does raptor for non tp6. This is used to ensure that we are not sending traffic to the live internet.

Flags: needinfo?(jmaher)
Blocks: 1547932
Priority: -- → P1

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #4)

I gave you an example, talos uses the same prefs as well and so does raptor for non tp6. This is used to ensure that we are not sending traffic to the live internet.

Who/what is we? The tests or Firefox (and several services)? I just wonder because it's only port 80 but not HTTPS too, which is mostly used those days.

Flags: needinfo?(jmaher)

we == firefox + tests

these settings have been around for 10 years and there has not been updating. I assume updating them for ssl would be a good thing.

Flags: needinfo?(jmaher)

So while investigating more it looks like that the proxy is only there so all the services whose preferences for remote URLs are modified in perf/user.js get routed to 127.0.0.1.

Note that for other test harnesses we do not have such a logic, but set the host as %(server)s in the preference value, and let the specific harness interpolate this value based on it's own locally running HTTPd instance.

I'm fairly sure that this should be adapted by any harness using those specific perf preferences. With that we won't need the proxy for port 80 nor for 443 at all.

Harnesses affected here are just Talos, and Raptor.

Joel, would that be ok with you?

Component: Raptor → General
Flags: needinfo?(jmaher)
Summary: "use_live_sites = true" has to disable the proxy server → Remove usage of proxy server from testing/profiles/perf/user.js

this might work out- will we cause a crash to the browser if it tries to access external sites? I believe that is the case already, but I want to confirm.

Lastly, to turn off services via prefs is difficult- we add or edit services often and they need to be kept up between all the profiles in testing/profiles- could you file a bug so we have one place to turn off all services?

Flags: needinfo?(jmaher)

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #8)

this might work out- will we cause a crash to the browser if it tries to access external sites? I believe that is the case already, but I want to confirm.

Correct. We only have to make sure that the environment variable MOZ_DISABLE_NONLOCAL_CONNECTIONS is set in automation for all of the performance jobs. I assume that this has already been done a while ago, even for Talos and Raptor. But I can check that again.

Lastly, to turn off services via prefs is difficult- we add or edit services often and they need to be kept up between all the profiles in testing/profiles- could you file a bug so we have one place to turn off all services?

Do you mean an extra folder under testing/profiles something like local which would contain all those entries, which are currently spread across various other sub folders?

Flags: needinfo?(jmaher)

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] from comment #9)

Do you mean an extra folder under testing/profiles something like local which would contain all those entries, which are currently spread across various other sub folders?

Note that this would allow some test harnesses to include the common and base preferences, but opt-out from using the force local settings. This is something Marionette based harnesses like firefox-ui-functional remote tests could benefit from in the future.

So both Talos and Raptor are using this environment variable. As such I just pushed a try build with the proxy settings removed for both Talos and Raptor but only Linux64. Not sure if there are tests only running on eg Windows.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=a5df2a1cb6defde321b55483394d871ed44b5c21

Lets see how much failures we see.

I think having our common profile preferences well defined so we have a single place for all external blocking/spoofing server preferences to live- a good idea.

afaik, windows has xperf and that is the only test unique to windows; osx shouldn't have any unique tests- so linux is a good sample.

Flags: needinfo?(jmaher)

As discussed with Joel on IRC the problem here is most likely the reference to the remove w3c logo. I replaced it with a local copy of it, and here is an updated try build for both the dromaeojs and g3 jobs:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=efb417eae51ec92beaceb46bf8eb6921670bffa6

Otherwise all other jobs are still green!

Assignee: nobody → hskupin
Status: NEW → ASSIGNED
Duplicate of this bug: 1547932

Just another try build with xperf job and raptor-youtube-playback, given that I was able to get rid of the horrible proxy workaround too.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=b5148309c279a3236f28f339aef927fa7116acdf

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #12)

I think having our common profile preferences well defined so we have a single place for all external blocking/spoofing server preferences to live- a good idea.

I filed bug 1553949 for that.

Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/2bdc4e2c82ae
[talos] Replace remote w3c image with local copy. r=jmaher
https://hg.mozilla.org/integration/autoland/rev/5b555225cd80
Remove usage of proxy server from testing/profiles/perf/user.js. r=jmaher
https://hg.mozilla.org/integration/autoland/rev/77898b7a7ea2
[raptor] Remove proxy workaround from raptor-youtube-playback tests. r=perftest-reviewers,stephendonner
Status: ASSIGNED → RESOLVED
Closed: 6 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla69
Regressions: 1554570

Looks like this harness change updated our baseline:
== Change summary for alert #21343 (as of Thu, 06 Jun 2019 14:09:42 GMT) ==

Improvements:

6% dromaeo_css windows10-64-shippable opt e10s stylo 14,369.78 -> 15,177.96

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=21343

You need to log in before you can comment on or make changes to this bug.