1582359 - FF does not read system proxy settings correctly on Ubuntu

Status: RESOLVED FIXED

(Core :: Networking, enhancement, P3)

Description

[laurent combe]

+++ This bug was initially created as a clone of Bug #1423497 +++

User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.0 Safari/537.36
Ubuntu 18.04

Steps to reproduce:

I have defined a system wide using ubuntu settings tool
looks like
user:password@proxy:8080

Actual results:

FF is unable to contact the proxy
(Chromium in the same environment can display web pages correctly using the proxy)

today i manually switch the firefox config beetween
no proxy (where i am at home)
manual settings (where i am at office)

Expected results:

FF should read the system proxy settings

Comment 1

[Junior [inactive]]

Did you mean that "Use system proxy setting" in Preferences>Network Settings>Settings doesn't work?

Comment 2

[laurent combe]

exactly

in fact if in ubuntu proxy settings i put
http://username:[e-mail]:8000
"Use system proxy setting" seems to work for FF
but by example evolution (the gnome mailer) does not like this way

if i put the same value without the protocol part (i.e http://)
username:[e-mail]:8000
FF does not find the proxy
but other apps yes

maybe FF may be smart
and detect if the protocol part is missing suppose by default "http://" ?

difficult to find a official doc for ubuntu proxy settings
difficult to see how FF decode the proxy settings and if it seems correct or not

Comment 3

[laurent combe]

i ask a question on ubuntu about my issue :
https://answers.launchpad.net/ubuntu/+source/ubuntu-docs/+question/684083

Comment 4

[Junior [inactive]]

We have many types of proxy: http, socks, ssl. I'm not sure if it's a good idea to default with http from arbitrary os.
Let's see what preference folks think.

Comment 5

[laurent combe]

1 - i suspect ubuntu define system proxy with
<user>:<password>@<proxy_host>:<port>
i open a question to clear this point :
https://answers.launchpad.net/ubuntu/+source/ubuntu-docs/+question/684083
(but no answer at this time)

2 - i suspect firefox expect the following in the proxy system setting
<protocol>://<user>:<password>@<proxy_host>:<port>

• can you confirm that ?

• after we can try to find a solution to handle this issue :
  i suggest to default to 'http' if the protocol is not specified
  (i know that there is many proxy types)

Comment 6

[:Gijs (he/him)]

(In reply to Junior [:junior] from comment #4)

We have many types of proxy: http, socks, ssl. I'm not sure if it's a good idea to default with http from arbitrary os.
Let's see what preference folks think.

I don't understand - how we interpret and use the system proxy URL is something networking code does, not preferences, so this is something for the networking folks to decide. The code seems to be at https://searchfox.org/mozilla-central/source/toolkit/system/unixproxy/nsUnixSystemProxySettings.cpp . From a very casual look, it looks like the code at https://searchfox.org/mozilla-central/rev/153feabebc2d13bb4c29ef8adf104ec1ebd246ae/toolkit/system/unixproxy/nsUnixSystemProxySettings.cpp#171-180 has done what it does for about 11 years. So presumably, either something changed here on the Linux/Ubuntu side, or Evolution doesn't know how to use an http proxy, or we've done things "wrong" for 11 years, in which case I'd expect other bugs on file. But I and the other frontend pref UI folks are not linux system experts, so I don't think we're the right people to help figure out which of those 3 things is the case.

Comment 7

[Junior [inactive]]

Thanks for pointing this out, :Gijs. I was looking for the UX's point of view, but this is out of Firefox's scope.
I'll raise this to team. Stay tuned.

Comment 8

[:Gijs (he/him)]

(In reply to Junior [:junior] from comment #7)

Thanks for pointing this out, :Gijs. I was looking for the UX's point of view, but this is out of Firefox's scope.

Ah, sorry. If you still think you need a UX perspective, #ux on slack is probably your best bet.

Comment 9

[laurent combe]

a similar question find on ubuntu forum :
https://askubuntu.com/questions/1061629/unable-to-setup-proxy-system-wide-in-kubuntu-18-04

for the code i suspect that the decoding function is nsUnixSystemProxySettings::SetProxyResultFromGSettings
https://searchfox.org/mozilla-central/rev/153feabebc2d13bb4c29ef8adf104ec1ebd246ae/toolkit/system/unixproxy/nsUnixSystemProxySettings.cpp#194-223
because my Firefox is influenced by the gsetting conf

if only i have a firefox with some trace activated to 'see' the decoding of gsetting proxy config
it will be easy to understand the issue ...

Comment 10

[Junior [inactive]]

Could I know your step to reproduce?
As I know we need to specify the protocol http/https/ftp in ubuntu for both UI and environment variable.

Comment 11

[laurent combe]

Could you find a reference an example when you need to specify protocol for UI ?

i ask to ubuntu but no answer ...
i don't find this information on ubuntu site
on the web all the exemples shows there is no protocol

it's not an ubuntu official but look at
https://www.serverlab.ca/tutorials/linux/administration-linux/how-to-configure-proxy-on-ubuntu-18-04/
and on the first image
https://www.serverlab.ca/wp-content/uploads/2018/04/ubuntu-18.04-proxy-settings.png
There is no protocol part

even the name of the gsetting field, 'host' suggest that there is no protocol part
org.gnome.system.proxy.http host <value>
org.gnome.system.proxy.http port <port>

to reply to your question 'step to reproduce'
my proxy is configured by my company and this context is specific (you can't access it)
i will try this week-end to launch a squid proxy on my laptop
and i hope to be able to give you a clear list of steps to reproduce

Comment 12

[Junior [inactive]]

It doesn't matter a lot for which proxy we use.
The thing is we rely on env variable http_proxy (or all-proxy), which should be auto filled http:// by ubuntu (haven't tested)

If the distribution of linux you use is not public, we don't have much room to move forward.

Comment 13

[Michal Novotny [:michal]]

(In reply to Junior [:junior] from comment #12)

It doesn't matter a lot for which proxy we use.
The thing is we rely on env variable http_proxy (or all-proxy), which should be auto filled http:// by ubuntu (haven't tested)

That's not true, see link in comment 9.

Comment 14

[Michal Novotny [:michal]]

(In reply to laurent combe from comment #2)

in fact if in ubuntu proxy settings i put
http://username:[e-mail]:8000
"Use system proxy setting" seems to work for FF
but by example evolution (the gnome mailer) does not like this way

If I understand correctly, it works if you specify http schema in the field that's labels "HTTP Proxy", right? That really shouldn't be needed. Just to make sure, could you please provide output of "gsettings list-recursively org.gnome.system.proxy" when the proxy doesn't work (i.e. without the schema)?

Comment 15

[laurent combe]

to resume :

first case : ubuntu proxy setup with: user:password@proxy_host
FF (with use proxy from system) do not success to catch the host, user and password.
In Ubuntu setup there is no other way to describe user and password for the proxy
but chromium (by example) success to find the proxy hosts
in ubuntu terminal the proxy env variable are correct

second case : ubuntu proxy setup with: proxy_host
FF success to catch the host, and ask for user and password to pass the proxy authentication
the proxy env variables are now missing user and password

• wget is not working (because it relies on env proxy variables)
• chromium catch the host, and ask for user and password to pass the proxy authentication
  In Ubuntu setup there is no other way to describe user and password for the proxy

this issue is at the frontier between Firefox and Ubuntu
but it's not a easy situation for end-users
your help is welcome

steps to reproduce :

i install a squid proxy on my laptop with Ubuntu 18.04 LTS
inspired from : https://stackoverflow.com/questions/3297196/how-to-set-up-a-squid-proxy-with-basic-username-and-password-authentication
$ sudo apt-get install squid
$ sudo apt-get install apache2-utils
(below my squid conf)
$ cat /etc/squid/squid.conf
auth_param basic program /usr/lib/squid3/basic_ncsa_auth /etc/squid/passwords
auth_param basic realm proxy
acl authenticated proxy_auth REQUIRED
http_access allow authenticated

Choose the port you want. Below we set it to default 3128.

http_port 3128

Ok now i configure a user for my quid proxy
say login: 'user' and password 'proxy_password'
and after i restart my squid:

htpasswd -c /etc/squid/passwords user

(it asks for password twice)
#service squid restart

ok now my proxy is running and ask authentication for browsing through it

i configure my Ubuntu proxy. below i list the value of m
$ gsettings list-recursively org.gnome.system.proxy
org.gnome.system.proxy use-same-proxy true
org.gnome.system.proxy mode 'manual'
org.gnome.system.proxy autoconfig-url ''
org.gnome.system.proxy ignore-hosts ['localhost', '127.0.0.0/8', '::1']
org.gnome.system.proxy.ftp host ''
org.gnome.system.proxy.ftp port 0
org.gnome.system.proxy.socks host ''
org.gnome.system.proxy.socks port 0
org.gnome.system.proxy.http host 'user:proxy_password@127.0.0.1'
org.gnome.system.proxy.http port 3128
org.gnome.system.proxy.http use-authentication false
org.gnome.system.proxy.http authentication-password ''
org.gnome.system.proxy.http authentication-user ''
org.gnome.system.proxy.http enabled false
org.gnome.system.proxy.https host 'user:proxy_password@127.0.0.1'
org.gnome.system.proxy.https port 3128

Now i open a terminal and check the environment :
$ env | grep proxy -i
HTTP_PROXY=http://user:proxy_password@127.0.0.1:3128/
https_proxy=http://user:proxy_password@127.0.0.1:3128/
http_proxy=http://user:proxy_password@127.0.0.1:3128/
no_proxy=localhost,127.0.0.0/8,::1
NO_PROXY=localhost,127.0.0.0/8,::1
HTTPS_PROXY=http://user:proxy_password@127.0.0.1:3128/

in this config wget is working
$ wget linuxfr.org
--2019-09-28 13:42:43-- http://linuxfr.org/
Connexion à 127.0.0.1:3128… connecté.
requête Proxy transmise, en attente de la réponse… 301 Moved Permanently
Emplacement : https://linuxfr.org/ [suivant]
--2019-09-28 13:42:43-- https://linuxfr.org/
Connexion à 127.0.0.1:3128… connecté.
requête Proxy transmise, en attente de la réponse… 200 OK

but if i launch firefox
error message with proxy

now if i omit in my ubuntu proxy conf the 'user' and 'proxy_password'
my env is now :
$ env | grep proxy -i
HTTP_PROXY=http://127.0.0.1:3128/
https_proxy=http://127.0.0.1:3128/
http_proxy=http://127.0.0.1:3128/
no_proxy=localhost,127.0.0.0/8,::1
NO_PROXY=localhost,127.0.0.0/8,::1
HTTPS_PROXY=http://127.0.0.1:3128/

and gsettings output is :
$ gsettings list-recursively org.gnome.system.proxy
org.gnome.system.proxy use-same-proxy true
org.gnome.system.proxy mode 'manual'
org.gnome.system.proxy autoconfig-url ''
org.gnome.system.proxy ignore-hosts ['localhost', '127.0.0.0/8', '::1']
org.gnome.system.proxy.ftp host ''
org.gnome.system.proxy.ftp port 0
org.gnome.system.proxy.socks host ''
org.gnome.system.proxy.socks port 0
org.gnome.system.proxy.http host '127.0.0.1'
org.gnome.system.proxy.http port 3128
org.gnome.system.proxy.http use-authentication false
org.gnome.system.proxy.http authentication-password ''
org.gnome.system.proxy.http authentication-user ''
org.gnome.system.proxy.http enabled false
org.gnome.system.proxy.https host '127.0.0.1'
org.gnome.system.proxy.https port 3128

in this context
wget is no longer working (407 authication required)
and FF knows how to contact the proxy but ask for a user/password

Comment 16

[laurent combe]

sorry
in the previous post
line beginning with a # are converted in big title (by mardown interpreter)
i do not find how to edit my post

Comment 17

[laurent combe]

so the problem is :

when the following key "org.gnome.system.proxy.http host"
is in the following format "<proxy_user>:<proxy_password>@<proxy_host>"

firefox do not succeed to determine the value of proxy_host
the lines of code involved is about :
https://searchfox.org/mozilla-central/rev/153feabebc2d13bb4c29ef8adf104ec1ebd246ae/toolkit/system/unixproxy/nsUnixSystemProxySettings.cpp#209-212

Comment 18

[Michal Novotny [:michal]]

(In reply to laurent combe from comment #17)

when the following key "org.gnome.system.proxy.http host"
is in the following format "<proxy_user>:<proxy_password>@<proxy_host>"

firefox do not succeed to determine the value of proxy_host

With this proxy settings the pacString at https://searchfox.org/mozilla-central/rev/23f836a71cfe961373c8bd0d0219ec60a64b3c8f/netwerk/base/nsProtocolProxyService.cpp#2123 contains:
PROXY username:password@host:port

Which then we fail to parse at https://searchfox.org/mozilla-central/rev/23f836a71cfe961373c8bd0d0219ec60a64b3c8f/netwerk/base/nsProtocolProxyService.cpp#1268

Comment 19

[Michal Novotny [:michal]]

Attachment: Bug 1582359 - Parse host:port correctly when PAC string includes credentials but scheme is not present, r?valentin

This patch fixes parsing host:port from PAC string when scheme is not present. The credentials are ignored just like in case it's parsed by NS_NewURI.

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/130708ffeae9
Parse host:port correctly when PAC string includes credentials but scheme is not present, r=valentin

Comment 21

[laurent combe]

thanks a lot
i'm very enthusiast to test it

today i have downloaded a nightly version
how to be sure my nightly contains your patch ?
my version : 71.0a1 (2019-10-02) (64 bits)
i will be waiting tomorrow to be sure

A little difficulty that has not be seen (not sure) :
in gsettings you can have :
key "org.gnome.system.proxy.http host"
with a value "<proxy_user>:<proxy_password>@<proxy_host>"

and later you can desactivate the proxy selecting "none"
but the 'host' key does not change it keeps its value, you have to look at 'mode' key

org.gnome.system.proxy mode 'none'
org.gnome.system.proxy.http host 'user:proxy_password@127.0.0.1'

in this case there is no proxy configured for the system
it's what i have on ubuntu. not tested on redhat ...

Comment 22

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/130708ffeae9

Comment 23

[laurent combe]

ok it works with firefox 71.0a1
14 days only ! thanks to michal and firefox team.

Comment 24

[Michal Novotny [:michal]]

(In reply to laurent combe from comment #21)

and later you can desactivate the proxy selecting "none"
but the 'host' key does not change it keeps its value, you have to look at 'mode' key

This should work as expected. I.e. we don't use proxy when mode is set to "none".

(In reply to laurent combe from comment #23)

ok it works with firefox 71.0a1

Thanks for confirming the fix.