1519159 - Temporary failure in name resolution attempting to use tooltool to download files from Bitbar containers

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Bob Clary [:bc] (inactive)]

+++ This bug was initially created as a clone of Bug #1499246 +++

Filing this to keep track of the issue and hopefully lead to a resolution...

Beginning at Tue, Oct 9, 12:15:26 with https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed,busted,exception,runnable&tier=1,2,3&group_state=expanded&searchStr=android-hw&revision=4a92019d1ad6be91a38aed8033b5b85d1654dd78 the android-hw tests began to experience frequent intermittent failures to fetch files on try jobs which are most frequently run.

https://treeherder.mozilla.org/logviewer.html#?job_id=204314059&repo=try&lineNumber=795

The first mozilla-central job where this occurred was at Tue, Oct 9, 09:24:03 with https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&resultStatus=testfailed%2Cbusted%2Cexception%2Crunnable&tier=1%2C2%2C3&group_state=expanded&searchStr=android-hw&selectedJob=204295629&revision=e96bcfe8669abdb7eaa9f034daba53d44d8c3e51

The first autoland job where this occured was at Tue, Oct 9, 09:29:58 with https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception%2Crunnable&tier=1%2C2%2C3&group_state=expanded&searchStr=android-hw&selectedJob=204270932&revision=1c93105605f888686118d498efacc64e92080a63

The first mozilla-inbound job where this occurred was at Tue, Oct 9, 08:04:39 with https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&resultStatus=testfailed%2Cbusted%2Cexception%2Crunnable&tier=1%2C2%2C3&group_state=expanded&searchStr=android-hw&selectedJob=204284415&revision=d49a5d674e007fc79beda9437889ca5a1eec4aa2

I made a slight change to log the exception when tooltool.py's fetch fails and found

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&author=bclary%40mozilla.com&group_state=expanded&fromchange=1f676fd17e982396b59472f9746a48e173a37575&selectedJob=207586835
https://treeherder.mozilla.org/logviewer.html#?job_id=207586835&repo=try&lineNumber=800

18:10:09 INFO - INFO - Attempting to fetch from 'https://tooltool.mozilla-releng.net/'...
18:10:19 INFO - INFO - ...failed to fetch 'linux64-minidump_stackwalk' from https://tooltool.mozilla-releng.net/
18:10:19 INFO - INFO - <urlopen error [Errno -3] Temporary failure in name resolution>
18:10:19 ERROR - ERROR - The following files failed: 'linux64-minidump_stackwalk'
18:10:19 ERROR - Return code: 1

Bitbar could not find any networking issues on their end which might be the cause but perhaps they didn't look in the right places.

I think I found the point where the change caused the problems: Bug 1487798 comment 11

As part of the mitigation attempts, I temporarily switched to using Ubuntu 18.04 but that ultimately did not fix the issue which implies to me that it was not an obsolete version of a library at fault. I had to switch back to Ubuntu 16.04 due to incompatible libraries which prevented minidump_stackwalk from working on 18.04.

I have worked around the problem by backing the tooltool_cache for needed files into the image created a Bitbar but that is a less than desirable solution.

Comment 1

[Rok Garbas [:garbas]]

:bc The problem in Bug 1487798 that the too old python was used. which version of python is used on on Ubuntu 18.04/16.04?

Comment 2

[Bob Clary [:bc] (inactive)]

The current 16.04 images are using 2.7.12. I don't have the 18.04 images available and the logs from my runs have all expired. 18.04 wasn't viable due to library conflicts with libcurl3 due to minidump_stackwalk iirc. What is the version used in the normal aws images? Do you have a recipe for the proper way to install the correct version on 16.04?

Comment 3

[Rok Garbas [:garbas]]

:bc i think 2.7.10 was the last version of python having this problems. are you able to get me a ssh login to the machine so I can try to debug it?

Comment 4

[Bob Clary [:bc] (inactive)]

Unfortunately not. I'll put you in contact with someone at Bitbar and perhaps they can help under your direction.