1085348 - Cache mock rpms with proxxy

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Carsten Book [:Tomcat]]

seems we have sometimes failures related to yum updates like

http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android/1413809414/mozilla-central-android-bm77-build1-build158.txt.gz

http://mockbuild-repos.pub.build.mozilla.org/releng/public/CentOS/6/x86_64/android-sdk16-r16-0moz2.i686.rpm: [Errno 12] Timeout on http://mockbuild-repos.pub.build.mozilla.org/releng/public/CentOS/6/x86_64/android-sdk16-r16-0moz2.i686.rpm: (28, 'Operation too slow. Less than 1 bytes/sec transfered the last 30 seconds')

http://mockbuild-repos.pub.build.mozilla.org/releng/public/CentOS/6/x86_64/android-sdk16-r16-0moz2.i686.rpm: [Errno 12] Timeout on http://mockbuild-repos.pub.build.mozilla.org/releng/public/CentOS/6/x86_64/android-sdk16-r16-0moz2.i686.rpm: (28, 'Operation too slow. Less than 1 bytes/sec transfered the last 30 seconds')

Comment 1

[Legacy TBPL/Treeherder Robot]

submit_timestamp: 2014-10-20T06:22:52
log: https://treeherder.mozilla.org/ui/logviewer.html#?repo=mozilla-central&job_id=505845
repository: mozilla-central
who: tomcat[at]mozilla[dot]com
machine: bld-linux64-spot-066
buildname: Android 2.3 mozilla-central build
revision: f2d7d694aae5

Comment 2

[Legacy TBPL/Treeherder Robot]

submit_timestamp: 2014-10-20T06:22:52
log: https://treeherder.mozilla.org/ui/logviewer.html#?repo=mozilla-central&job_id=505891
repository: mozilla-central
who: tomcat[at]mozilla[dot]com
machine: bld-linux64-spot-1046
buildname: b2g_mozilla-central_linux64_gecko build
revision: f2d7d694aae5

Comment 3

[Legacy TBPL/Treeherder Robot]

submit_timestamp: 2014-10-20T06:22:52
log: https://treeherder.mozilla.org/ui/logviewer.html#?repo=mozilla-central&job_id=505846
repository: mozilla-central
who: tomcat[at]mozilla[dot]com
machine: bld-linux64-spot-1036
buildname: Android 2.3 Debug mozilla-central build
revision: f2d7d694aae5

Comment 4

[Chris Cooper [:coop] (he/him)]

The 3 reported instances are all in use1, but there's nothing anomalous in the network connection to AWS on Monday, Oct 20:

http://netops2.private.scl3.mozilla.com/smokeping/sm.cgi?target=Datacenters.RELENG-SCL3.nagios1-releng-use1

I'll check the mockbuild-repos host(s) (they're behind the load balancer) to see if there were any issues on Monday.

Mock is going to be pulling a lot of data. Even with proxxy, it might make sense to have a repo host in AWS if this becomes a chronic issue.

Comment 5

[Chris Cooper [:coop] (he/him)]

If this happens again, we can re-open.

Comment 6

[Pete Moore [:pmoore][:pete]]

http://ftp.mozilla.org/pub/mozilla.org/spidermonkey/tinderbox-builds/mozilla-inbound-linux64-debug/mozilla-inbound_linux64-debug_spidermonkey-rootanalysis-bm70-build1-build221.txt.gz

Comment 7

[Pete Moore [:pmoore][:pete]]

Occured around: 2014-10-31 02:02:20.187536 (presumably PDT)

Comment 8

[Pete Moore [:pmoore][:pete]]

http://mockbuild-repos.pub.build.mozilla.org/mirrors/centos/6/latest/os/x86_64/ConsoleKit-0.4.1-3.el6.x86_64.rpm: [Errno 14] PYCURL ERROR 7 - "couldn't connect to host"
Trying other mirror.


Error Downloading Packages:
  ConsoleKit-0.4.1-3.el6.x86_64: failure: ConsoleKit-0.4.1-3.el6.x86_64.rpm from centos6: [Errno 256] No more mirrors to try.

Comment 9

[Chris Cooper [:coop] (he/him)]

(In reply to Pete Moore [:pete][:pmoore] from comment #8)
> http://mockbuild-repos.pub.build.mozilla.org/mirrors/centos/6/latest/os/
> x86_64/ConsoleKit-0.4.1-3.el6.x86_64.rpm: [Errno 14] PYCURL ERROR 7 -
> "couldn't connect to host"
> Trying other mirror.
> 
> 
> Error Downloading Packages:
>   ConsoleKit-0.4.1-3.el6.x86_64: failure: ConsoleKit-0.4.1-3.el6.x86_64.rpm
> from centos6: [Errno 256] No more mirrors to try.

Weird that it only failed to connect for one rpm out of 250. Was our timing really that lucky/unlucky, or is there local caching happening here?

Comment 10

[:kanban-engops]

A Pivotal Tracker story has been created for this Bug: https://www.pivotaltracker.com/story/show/81854234

Comment 11

[:kanban-engops]

A Pivotal Tracker story has been created for this Bug: https://www.pivotaltracker.com/story/show/82064760

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

It doesn't look like this is going through proxxy -- that would probably be the best fix here, right?

Comment 13

[Legacy TBPL/Treeherder Robot]

submit_timestamp: 2014-11-06T14:43:52
log: https://treeherder.mozilla.org/ui/logviewer.html#?repo=fx-team&job_id=1120883
repository: fx-team
who: wkocher[at]mozilla[dot]com
machine: bld-linux64-spot-1009
buildname: Android 2.3 Debug fx-team build
revision: f2237c6455ab

Comment 14

[Legacy TBPL/Treeherder Robot]

submit_timestamp: 2014-11-06T14:43:52
log: https://treeherder.mozilla.org/ui/logviewer.html#?repo=fx-team&job_id=1120893
repository: fx-team
who: wkocher[at]mozilla[dot]com
machine: bld-linux64-spot-009
buildname: Linux x86-64 fx-team debug static analysis build
revision: f2237c6455ab

Comment 15

[Chris Cooper [:coop] (he/him)]

Another failure today:

http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/fx-team-linux64-st-an-debug/1415311758/fx-team-linux64-st-an-debug-bm71-build1-build296.txt.gz

(In reply to Dustin J. Mitchell [:dustin] from comment #12)
> It doesn't look like this is going through proxxy -- that would probably be
> the best fix here, right?

That would certainly help, yes.