Closed Bug 916724 Opened 12 years ago Closed 12 years ago

Wrong proxy settings for Ubuntu 13.04 machines are causing no connection to various hosts

Categories

(Mozilla QA Graveyard :: Infrastructure, defect)

Product:
Mozilla QA Graveyard
Component:
Infrastructure
Platform:
All
Linux
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: whimboo, Assigned: whimboo)

References

Details

As seen by end of last week our Ubuntu 13.04 machines have issues trying to find updates. Here an example: http://mm-ci-master.qa.scl3.mozilla.com:8080/job/ondemand_update/23115/console Tested channel is 'esrtest' with Firefox 17.0.5esr.
I did a couple of testruns on that affected box but I was not able to recreate the situation. It only failed when triggered directly through Jenkins. That's strange. I will do another Jenkins run to see if something has been changed.
Actually I have no idea what's causing the problem. As mentioned before those jobs only fail when triggered by Jenkins itself. The only effect here could be environmnet variables which get pushed to the slaves. Sadly we don't see what's in use as long as the following issue hasn't been fixed: https://github.com/mozilla/mozmill-ci/issues/191 Dave will work on that so that we can push this feature to production after 1 day of testing and the 24.0 version released.
Whiteboard: [blocked by CI issue 191]
Not blocked anymore. I will check what has been caused this. I have seen a couple of failed testruns over the weekend: http://mm-ci-master.qa.scl3.mozilla.com:8080/job/mozilla-esr17_update/1076/ http://mm-ci-master.qa.scl3.mozilla.com:8080/job/mozilla-esr17_update/1074/
Whiteboard: [blocked by CI issue 191]
The failures here might be related to the proxy given that other CI instances can run updates successfully. I will have to analyze the HTTP log.
Ok, so this is indeed a proxy setting issue. Whether the HTTP_PROXY nor FTP_PROXY environment variables are set: http://mm-ci-master.qa.scl3.mozilla.com:8080/job/mozilla-esr17_update/1076/injectedEnvVars/? I will update those machines now.
Adrian, the node mm-ub-1304-32-3 didn't came back when I rebooted it. I need your help. Any idea why that happened again? It's strange given that we have had it already a couple of times.
Flags: needinfo?(afernandez)
mm-ub-1304-32-3 back online. Apparently init fails at a proper shutdown as the "modem manager" process fails to obtain the system bus. Init then goes into single user mode. Not sure why a modem manager is even running but I would suggest to disable it.
Flags: needinfo?(afernandez)
We removed the modemmanager package from Ubuntu so all is working fine now again. I will continue investigating this bug tomorrow.
modemmanager was also removed from the four Ubuntu templates; Ubuntu_12.04_x86 Ubuntu_12.04_x86-64 Ubuntu_13.04_x86 Ubuntu_13.04_x86-64
This has to be definitely be related to the proxy settings. Starting Firefox via a testrun initialized from Jenkins, I'm not able to even connect to our mm-ci-master host. So Jenkins most likely injects or removes proxy environment settings.
Injected proxy environment variables Successful Ubuntu 12.04 testrun: ftp_proxy http://proxy.dmz.scl3.mozilla.com:3128/ http_proxy http://proxy.dmz.scl3.mozilla.com:3128/ https_proxy http://proxy.dmz.scl3.mozilla.com:3128/ no_proxy localhost,127.0.0.1,localaddress,.localdomain.com,mm-ci-master,.scl3.mozilla.com,.phx1.mozilla.com Failing Ubuntu 13.04 testrun: HTTPS_PROXY http://proxy.dmz.scl3.mozilla.com:3128/ ftp_proxy http://proxy.dmz.scl3.mozilla.com:3128/ http_proxy http://proxy.dmz.scl3.mozilla.com:3128/ no_proxy localhost,127.0.0.1,localaddress,.localdomain.com,10.0.0.0/8,.scl3.mozilla.com,.scl3.mozilla.com I have corrected the settings on 13.04-32-3 and will run a testrun now.
This were indeed the wrong proxy settings from /etc/environment which included all entries in uppercase letters. All of them will have to be removed and only the lowercase entries should remain. I will do further tests for more recent releases of Firefox if it stays green.
It's not only that update tests are affected by that. All tests which want to access certain sites will fail because of it.
Summary: Ubuntu 13.04 machines are failing in checking for updates step for Firefox 17.0ESR release → Wrong proxy settings for Ubuntu 13.04 machines are causing no connection to various hosts
(In reply to Henrik Skupin (:whimboo) from comment #15) > I will check if the proposed solution from the MANA page will work: > https://mana.mozilla.org/wiki/display/SECURITY/ > Configuring+Your+Servers+to+Use+Proxies#ConfiguringYourServerstoUseProxies- > Linux Ok, so we cannot follow this suggestion because the uppercase environment variables are causing us massive problems. Also I don't that we don't need them here. They are getting already defined in the system preferences, and are necessary for Firefox itself. It might be a bug in Firefox 17.0 and earlier that it was not working correctly, but as long as we have to run tests for 17.0 we might have to live with this workaround. So we should use the same settings as we have for Ubuntu 12.04. Which is lowercase letters only. Adrian, can you please convert the Ubuntu templates to VMs, so that I can make this change. Beside that I would also like to find out how to disable the Backup notifications which appear occasionally.
Flags: needinfo?(afernandez)
Following Ubuntu "templates" are back online: Ubuntu_12.04_x86 10.22.73.242 Ubuntu_12.04_x86-64 10.22.73.243 Ubuntu_13.04_x86 10.22.73.238 Ubuntu_13.04_x86-64 10.22.73.222
Flags: needinfo?(afernandez)
(In reply to Adrian Fernandez [:Aj] from comment #17) > Ubuntu_12.04_x86 10.22.73.242 > Ubuntu_12.04_x86-64 10.22.73.243 > Ubuntu_13.04_x86 10.22.73.238 > Ubuntu_13.04_x86-64 10.22.73.222 All done. You can convert them back to templates. With that we are done on this bug.
Ups, hit return to early. The last machine is still running an update. So i will report back once it is done.
Ok, all done now. VMs can be converted back to templates. Thanks.
Flags: needinfo?(afernandez)
VMs converted back to templates.
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Flags: needinfo?(afernandez)
Resolution: --- → FIXED
Product: Mozilla QA → Mozilla QA Graveyard
You need to log in before you can comment on or make changes to this bug.