1514391 - Add ability to download files from tooltool from within the image running at bitbar

Status: RESOLVED FIXED

(Testing :: Raptor, enhancement)

Description

[Robert Wood [:rwood]]

Raptor pageload tests running on android geckoview (Bug 1506912) use tooltool to download everything required to use mitmproxy - including the mitmproxy tool and archive of recordings.

The actual recordings ZIP downloaded from tooltool is different for each Raptor test; and more recording ZIPs will be added in the future as the pageload tests are expanded.

Currently the Raptor job cannot access tooltool when running on bitbar:

21:43:45     INFO -  raptor-mitmproxy INFO - Attempting to fetch from 'https://tooltool.mozilla-releng.net/'...
21:44:05     INFO -  raptor-mitmproxy INFO - ...failed to fetch 'mitmproxy-2.0.2-linux.tar.gz' from https://tooltool.mozilla-releng.net/
21:44:05    ERROR -  raptor-mitmproxy Traceback (most recent call last):
21:44:05     INFO -  raptor-mitmproxy   File "/builds/worker/workspace/mozharness/external_tools/tooltool.py", line 480, in fetch_file
<... more stack...>
21:44:05     INFO -  raptor-mitmproxy   File "/usr/lib/python2.7/urllib2.py", line 1198, in do_open
21:44:05     INFO -  raptor-mitmproxy     raise URLError(err)
21:44:05     INFO -  raptor-mitmproxy URLError: <urlopen error [Errno -3] Temporary failure in name resolution>
21:44:05     INFO -  raptor-mitmproxy ERROR - The following files failed: 'mitmproxy-2.0.2-linux.tar.gz'

Raptor also uses the 'certutil' tool to create the geckoview certificate database - so the bitbar image will also need access to hostutils also please. Thanks!

Comment 1

[Bob Clary [:bc] (inactive)]

The only sure fire way of working around the networking issue with bitbar and tooltool that I've found so far is to bake the downloads into the tooltool cache in the image when it is created. Apart from that maybe we can fake the hostname / ip in a hosts file or something.  I spent a lot of time trying to work around this back in October so I can't guarantee a quick resolution except for the tooltool cache approach.

Comment 2

[Robert Wood [:rwood]]

As discussed on IRC, I provided :bc with the tooltool manifests for the mitmproxy binary (linux) and the first set of mobile recordings archive (tp6m-1); to be added into :bc's docker cache.

:bc, question: Raptor downloads and unpacks the binary and recordings to: obj.../testing/raptor. Will you be able to put them there? Or will I need to copy them from somewhere else? Thanks. I'll need to add a patch in Raptor to expect the binary and recordings to already exist instead of downloading them on tooltool.

Comment 3

[Bob Clary [:bc] (inactive)]

I don't yet have the manifests. Where are they?

If it uses tooltool.py, then you don't have to do anything. I populate the cache in the image and when you call tooltool to download the files, they will be found in the cache and loaded from there without having to hit the network.

Comment 4

[Bob Clary [:bc] (inactive)]

Attachment: mozilla-docker-20181220T124334-public.zip

Bitbar image definition with new manifests.

Comment 5

[Bob Clary [:bc] (inactive)]

Dockerfile version 20181220T124334 appears to be working fine. I'll file a different bug on the networking issues so that hopefully some day we won't have to cache these things in the images.

Comment 6

[Bob Clary [:bc] (inactive)]

filed bug 1519159 on the underlying issue.