949427 - Flashplugin-installer fails to update on Ubuntu machines due to broken proxy settings

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Infrastructure, defect)

Description

[Andreea Matei [:AndreeaMatei]]

We found this issue on our Ubuntu machines, where a dialog pops up stating the update for flashplugin-installer has failed and presents 2 options. To run the update at that point, which will open a terminal and download+install the update (then the terminal closes, but the popup remains open) or a close button which will just postpone the update. It will appear later again.

I think this dialog interferes with our testruns, as I saw when we had aborted jobs, a Firefox window was open (or 2 sometimes) and also this dialog. After I manually install it, the testrun goes well.

Maybe there's a network issue and we can't always download the update, but it's hard to think this happens so often on so many  machines (today I checked all of them and about 70% I had to install it myself).

Should we disable this check for update and make it a priority to update it ourselves from time to time?

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Do you have a screenshot of this popup? Is it an OS level popup or browser? We should definitely find a way to disable these update checks and perform them manually as a maintenance task.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

This window is only visible after a system update which I performed lately. I thought that when we go through apt-get directly this window doesn't pop-up. Via the console output the flash player got installed. So I wonder why we still see it. We might have to run dpkg after apt-get to download and install the latest flash player.

Comment 3

[Andreea Matei [:AndreeaMatei]]

Attachment: Screenshot.png

It's a OS window, for updates. It's also strange that the window still remains open, even after we click "Run this action" and the update is installed successfully.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Some helpful information can be found here:
http://ubuntuforums.org/showthread.php?t=2134888

I will have a look what we can do here.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

So it looks like that the problem here is based on broken proxy settings. At least on the Ubuntu 12.04 32bit node on staging I see the following:

$ cat /etc/apt/apt.conf
Acquire::http::proxy "http://proxy.dmz.scl3.mozilla.com:3128/";
Acquire::https::proxy "https://proxy.dmz.scl3.mozilla.com:3128/";
Acquire::ftp::proxy "ftp://proxy.dmz.scl3.mozilla.com:3128/";

The proxy does not support ftp:// or https://. So it should only be http://. Also those settings should not be present in that file but in /etc/apt/apt.conf.d/95proxies.

$ cat /etc/apt/apt.conf.d/95proxies
Acquire::http::proxy "http://proxy.dmz.scl3.mozilla.com:3128/";
Acquire::https::proxy "http://proxy.dmz.scl3.mozilla.com:3128/";
Acquire::ftp::proxy "http://proxy.dmz.scl3.mozilla.com:3128/";

To ensure the boxes don't come up again, please also run the following command so we can be sure the installer doesn't popup the window again.

$ sudo rm -f /var/lib/update-notifier/package-data-downloads/flashplugin-installer.failed 

For safety also restart the nodes.

Cosmin, would you be able to cover this given that you are duty this week?

Comment 6

[Cosmin Malutan, [:cosmin-malutan]]

Sure, I will take this.

Comment 7

[Cosmin Malutan, [:cosmin-malutan]]

I removed the lines from /etc/apt/apt.conf, checked /etc/apt/apt.conf.d/95proxies, run the command:
>sudo rm -f /var/lib/update-notifier/package-data-downloads/flashplugin-installer.failed 
After restart I had to connect via ssh to restart the lightdm (I couldn't connect via vnc)
mm-ub-1204-32-3 mm-ub-1204-32-4
mm-ub-1204-64-3 mm-ub-1204-64-4
mm-ub-1310-32-3 mm-ub-1310-32-4
mm-ub-1310-64-3 mm-ub-1310-64-4
I will update the rest of the nodes latter, right now endurance testruns are running.

Comment 8

[Cosmin Malutan, [:cosmin-malutan]]

I updated the rest of the nodes too.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

This is not fixed and we frequently see this when new updates are available for Flash. So I did a bit of investigation and figured out what's wrong. The actual download of the flash archive does not happen via apt but is a normal wget/curl command. So the proxy settings affected here are not the ones for apt, but inside the environment/profile config file.

When updating Flash you will run apt with sudo. Sadly in that mode no proxy environment variables are present. That means the download of the flash package via HTTPS fails because the server cannot be reached. The result is that the new Flash version is not installed, and the installer tries it again later by bringing up the update dialog. So we should fix our proxy settings to have all the env variables present when running with sudo.

I will take this.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

After a bit of searching I found the following blog post:
http://blog.troyastle.com/2009/11/proxy-environment-variables-and-sudo-in.html

As you can see sudo will automatically strip nearly all of the set environment variables when being called. We could update the sudoers file to let the proxy config pass-through, but that would require that we have to do it on all the nodes.

A better solution for us is to call apt-get via 'sudo -E apt-get'. This will keep all environment variables intact, and the additional download of the real flash package will succeed.

I have updated the Mana documentation for installing Flash, and running upgrades on Ubuntu. I will now run the steps on the Ubuntu staging machines. If they do not show the dialog anymore tomorrow, I will also fix all production machines.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

So I did this for staging for now. I will revisit tomorrow and if no dialogs are up on staging machines, we can safely fix production.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+1]]

It happened again on all the boxes, which I updated as described above. So I did a bit of investigation.

As listed on the following page, there is the file /var/lib/update-notifier/package-data-downloads/flashplugin-installer.failed which is causing the dialog to appear.

http://askubuntu.com/questions/317765/how-do-i-stop-failure-to-download-extra-data-files-error-with-pepflashpluginin

So I did a datetime check for this file, and have seen that the last update was exactly when cron.daily is running. So I checked /etc/cron.daily/update-notifier-common and it contains:

> # Try to rerun any package data downloads that failed at package install time.
> /usr/lib/update-notifier/package-data-downloader

So by running this tool via sudo I can bring the dialog up. The question is why this is happening, because the installation was successful. I will dig more into it and check when the .failed file is getting created. When we can prevent this the dialog should not show up anymore.

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+1]]

It's very strange. Even with the .failed file not being present after a successful installation of Flash the above tool still tries to download the extra data. So whether this tool is broken, or it looks for another file too.

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+1]]

As of the specs document: https://wiki.ubuntu.com/Specs/UpdateNotifierPackageDataDownloader

> Packages request data downloads by shipping a config file in the /usr/share/package-data-downloads/
> directory. The filename of the config file should preferably match the binary package name. 

Also:

> Stamp files are created in /var/lib/update-notifier/package-data-downloads/ to track the status of the
> downloads, using the same name as the config file from /usr/share/package-data-downloads/ with an 
> optional extension

Such a file does not exist after a successful installation of Flash, even with the triggers for update-notifier-common are processed successfully. Looks like that this status file is not successfully written out to the folder. So a cron.daily job run this download again, and creates the status file.

I will have to file a bug for that against the Ubuntu flashplugin-installer package.

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+1]]

For now lets use the following commands:

Installation:
sudo -E apt-get install flashplugin-installer && sudo /usr/lib/update-notifier/package-data-downloader

Update (if Flash was updated):
sudo apt-get update && sudo -E apt-get dist-upgrade && sudo /usr/lib/update-notifier/package-data-downloader

For the initial installation and an upgrade (if an Flash update is in the queue) it will trigger the download twice, but at least the second run sets the status file correctly. No update notifier will then complain about not being able to download the Flash packages.

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Ubuntu bug on file is now:
https://bugs.launchpad.net/ubuntu/+source/flashplugin-nonfree/+bug/1391347

Comment 17

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I have updated all the Ubuntu 14.04 machines on production as given above. Lets see if that helps. We will know more tomorrow.

Comment 18

[Andrei Eftimie]

Nice investigation and find Henrik.

Great to have this finally fixed!

Comment 19

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Thanks Andrei! I re-checked all the machines and there is no single update notifier dialog open. So I can go ahead and apply the changes to all the remaining boxes.

Comment 20

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I also updated the remaining Ubuntu 12.04 and 13.10 boxes on production. So none of them should bring up this dialog anymore.

I think we can call this bug fixed.