1050268 - Add gnome proxy settings for Ubuntu

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Puppet, task)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

As seen on bug 1043994 we have proxy issues in combination with Jenkins. For details see the other bug. Here the abstract what's conflicting for our systems:

* Jenkins supports only upper OR lowercase environment variables. It does not pass through both
* On the shell all tools AFAICS use the lowercase env variables

Given that Jenkins only passes through the lower OR uppercase env variable we see failures on the shell OR in Firefox.

To prevent this we decided to do the following on our existing Mozmill boxes in the past:

* We only expose the lowercase env variables for the shell
* We modified the gnome network settings for the proxy details

With that the shell (python) tools are working as expected, and Firefox and other GUI applications can retrieve the settings from Gnome.

Dustin, is that something we could also do for puppet? Or do you have concerns in removing the uppercase versions? Maybe we need special tweaking for CentOS. Also we would have to figure out how to set the proxy settings via gsettings.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The following settings we would have to set for the GUI applications:

# gsettings list-keys org.gnome.system.proxy
use-same-proxy
mode
autoconfig-url
ignore-hosts

# gsettings list-keys org.gnome.system.proxy.http
host
port
use-authentication
authentication-password
authentication-user
enabled

It would be similar to the method we do on OS X.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Recapping our irc convo, I tihnk adding gnome proxy settings sounds like a fine and good idea.

But *removing* the uppercase env vars seems like a bad idea, as I'm sure we'll find a tool that expects the uppercase vars (either now or sometime down the road), which then doesn't work.

This is a Jenkins bug, but it's been open for 1.5 years so it's unlikely to be closed soon.

One option may be to run either the jenkins client through a wrapper that removes the uppercase env vars.  That would limit the issue above to only Jenkins jobs.  Another option is to cause Jenkins to run every command through a wrapper script that sets all of the env vars appropriately.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Yeah, I will leave the uppercase env variables as you said. Maybe Firefox doesn't need those anymore when the gnome proxies have been set. It's something I will test tomorrow.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

gsettings set org.gnome.system.proxy mode 'manual'
gsettings set org.gnome.system.proxy.http enabled true
gsettings set org.gnome.system.proxy.http host 'proxy.dmz.scl3.mozilla.com'
gsettings set org.gnome.system.proxy.http port 3128

gsettings set org.gnome.system.proxy use-same-proxy true
gsettings set org.gnome.system.proxy ignore-hosts ['localhost', '127.0.0.1', 'localaddress', '.localdomain.com', '10.0.0.0/8', '.scl3.mozilla.com', '.phx1.mozilla.com', '.mozqa.com', 'mm-ci-master', 'mm-ci-staging', 'tps-ci-production', db1', 'puppet', 'repos']

I will experiment with those settings, and check if those are working.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

We have a bug in Firefox which doesn't let us access FQDN hosts even with a wildcard specified in ignore-hosts. Firefox simply ignores that, while other browsers are working fine. I filed bug 1050763 for that.

But that bug should not stop any work here. We only have to ensure to access any CI master via the pure host name.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Ok, so on bug 1050763 we figured out that we will have to use wildcards like *.scl3.mozilla.com for the gnome ignore-hosts list. Thanks Karl for the feedback.

But it still seems that the proxy settings have to be set by the individual user. Given that puppet runs under root, I'm not sure what's the best to apply those settings.

Karl, is there a magic feature, which I can use to set the proxy settings globally? So that it applies to all users on such a system?

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

What does "by the individual user" entail?  Puppet can run scripts as any uid, if that helps, and can also set environment variables.  If it requires being run in some kind of GNOME or dbus context, that might be harder..

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+2]]

When I run 'gsettings %command%' as root, it is only set for root. So if we want to set it for the builder, we would have to run 'su - %builder% -c gsettings'. But at this point the webproxy module would have to know the user. And that is not something I want to expose from that module. That's why a global setting of proxy settings for gnome would be preferred from my side. Maybe you have an another idea. But I would like to hear Karl's feedback given that he knows way more about all that.

Comment 9

[Karl Tomlinson (:karlt)]

GSettings settings have defaults, so it is possible to set these, but I don't know whether it is easy.  The only way I know is to modify the gsettings-desktop-schema package to have the desired defaults and install/replace the package.

Environment variables may be easier to use.  IIRC something was fixed so that these would semi-work even when gsettings-desktop-schemas was installed.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Karl Tomlinson (:karlt) from comment #9)
> Environment variables may be easier to use.  IIRC something was fixed so
> that these would semi-work even when gsettings-desktop-schemas was installed.

So we have all the environment variables present, which include no_proxy and NO_PROXY. But when I do not specify anything for ignore-hosts, Firefox does not automatically fallback to the environment variables for no_proxy/NO_PROXY. So not sure what to do here.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+2]]

It looks like that override files can be used to define new defaults without having to change the gsettings schema files. It can be done via an ini file, which has to be put into the same folder as the schema xml file. Details here (sorry I only found this in German): http://www.florian-diesch.de/doc/gnome3/standardeinstellungen-in-gsettings-aendern.html

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Ok, I got it working by placing a 99_proxy.gschema.override file into /usr/share/glib-2.0/schemas. After compiling the schemas with glib-compile-schemas the builder user gets the new default values immediately.

Here how this file looks like:

# cat /usr/share/glib-2.0/schemas/99_proxy.gschema.override 
[org.gnome.system.proxy]
mode = 'manual'
ignore-hosts = ['localhost', '127.0.0.1', 'localaddress', '.localdomain.com', '10.0.0.0/8', '.scl3.mozilla.com', '.phx1.mozilla.com', '.mozqa.com', 'mm-ci-master', 'mm-ci-staging', 'tps-ci-production', 'db1', 'puppet', 'repos']
use-same-proxy = true

[org.gnome.system.proxy.http]
enabled = true
host = 'proxy.dmz.scl3.mozilla.com'
port = 3128


It's interesting that here I don't have to use a wildcard for the subdomain issue as previously pointed out. Not sure why.

In case we want to remove the new default settings for the proxy, we only have to delete this file and re-compile the folder. That's all.

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: gnome_proxy v1

That correctly sets the proxy settings for gnome, and removes them if no proxy has been specified. It's way cheaper than on OS X, where we have to run multiple commands.

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8471478 [details] [diff] [review]
gnome_proxy v1

Review of attachment 8471478 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good with the addition on an include!

::: modules/web_proxy/manifests/environment.pp
@@ +4,5 @@
>  
>  class web_proxy::environment {
> +    $proxy_enabled = $web_proxy::host != ""
> +
> +    if $proxy_enabled {

I like this - it reads much better!

::: modules/web_proxy/manifests/gui.pp
@@ +29,5 @@
> +            file {
> +                "${schema_dir}/99_puppet_webproxy.gschema.override" :
> +                    ensure => $proxy_enabled ? { true => present, default => absent },
> +                    owner => "$users::root::username",
> +                    group => "$users::root::group",

If you reference users::root, you should include it.  I see it's used for Darwin, too.  The risk is that a reordering of manifests might load this class before users::root, in which case those variables will be undefined.

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: gnome_proxy v2

Good catch. I totally missed that all the time. Will remember that for the future. Here the updated version with the include.

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+2]]

http://hg.mozilla.org/qa/puppet/rev/e35bc82bc989 (production)
http://hg.mozilla.org/qa/puppet/rev/e860f804a921 (default)