986298 - mock builds need to use package mirrors

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=36477935&tree=Mozilla-Inbound

shows

mock_mozilla -r mozilla-centos6-i386 --install autoconf213 mozilla-python27 zip mozilla-python27-mercurial git ccache glibc-static libstdc++-static perl-Test-Simple perl-Config-General gtk2-devel libnotify-devel yasm alsa-lib-devel libcurl-devel wireless-tools-devel libX11-devel libXt-devel mesa-libGL-devel gnome-vfs2-devel mpfr xorg-x11-font imake ccache wget gcc472_0moz1 gcc473_0moz1 freetype-2.3.11-6.el6_2.9 freetype-devel-2.3.11-6.el6_2.9 gstreamer-devel gstreamer-plugins-base-devel
 in dir /builds/slave/m-in-linux32_g-000000000000000/build (timeout 2700 secs)
 watching logfiles {}
 argv: mock_mozilla -r mozilla-centos6-i386 --install autoconf213 mozilla-python27 zip mozilla-python27-mercurial git ccache glibc-static libstdc++-static perl-Test-Simple perl-Config-General gtk2-devel libnotify-devel yasm alsa-lib-devel libcurl-devel wireless-tools-devel libX11-devel libXt-devel mesa-libGL-devel gnome-vfs2-devel mpfr xorg-x11-font imake ccache wget gcc472_0moz1 gcc473_0moz1 freetype-2.3.11-6.el6_2.9 freetype-devel-2.3.11-6.el6_2.9 gstreamer-devel gstreamer-plugins-base-devel
 environment:
  CCACHE_HASHDIR=
  CVS_RSH=ssh
  G_BROKEN_FILENAMES=1
  HISTCONTROL=ignoredups
  HISTSIZE=1000
  HOME=/home/cltbld
  HOSTNAME=bld-linux64-spot-086.build.releng.use1.mozilla.com
  LANG=en_US.UTF-8
  LESSOPEN=|/usr/bin/lesspipe.sh %s
  LOGNAME=cltbld
  MAIL=/var/spool/mail/cltbld
  PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin
  PWD=/builds/slave/m-in-linux32_g-000000000000000/build
  SHELL=/bin/bash
  SHLVL=1
  TERM=linux
  TMOUT=86400
  USER=cltbld
  _=/tools/buildbot/bin/python
 using PTY: False
INFO: mock_mozilla.py version 1.0.3 starting...
State Changed: init plugins
INFO: selinux disabled
State Changed: start
Mock Version: 1.0.3
INFO: Mock Version: 1.0.3
State Changed: lock buildroot
INFO: installing package(s): autoconf213 mozilla-python27 zip mozilla-python27-mercurial git ccache glibc-static libstdc++-static perl-Test-Simple perl-Config-General gtk2-devel libnotify-devel yasm alsa-lib-devel libcurl-devel wireless-tools-devel libX11-devel libXt-devel mesa-libGL-devel gnome-vfs2-devel mpfr xorg-x11-font imake ccache wget gcc472_0moz1 gcc473_0moz1 freetype-2.3.11-6.el6_2.9 freetype-devel-2.3.11-6.el6_2.9 gstreamer-devel gstreamer-plugins-base-devel
ERROR: Command failed: 
 # ['/usr/bin/yum', '--installroot', '/builds/mock_mozilla/mozilla-centos6-i386/root/', 'install', 'autoconf213', 'mozilla-python27', 'zip', 'mozilla-python27-mercurial', 'git', 'ccache', 'glibc-static', 'libstdc++-static', 'perl-Test-Simple', 'perl-Config-General', 'gtk2-devel', 'libnotify-devel', 'yasm', 'alsa-lib-devel', 'libcurl-devel', 'wireless-tools-devel', 'libX11-devel', 'libXt-devel', 'mesa-libGL-devel', 'gnome-vfs2-devel', 'mpfr', 'xorg-x11-font', 'imake', 'ccache', 'wget', 'gcc472_0moz1', 'gcc473_0moz1', 'freetype-2.3.11-6.el6_2.9', 'freetype-devel-2.3.11-6.el6_2.9', 'gstreamer-devel', 'gstreamer-plugins-base-devel']
http://repos/repos/yum/mirrors/centos/6/latest/os/i386/repodata/repomd.xml: [Errno 14] PYCURL ERROR 6 - "Couldn't resolve host 'repos'"
Trying other mirror.
http://repos/repos/yum/mirrors/centos/6/latest/updates/i386/repodata/repomd.xml: [Errno 14] PYCURL ERROR 6 - "Couldn't resolve host 'repos'"
Trying other mirror.
http://repos/repos/yum/releng/public/CentOS/6/i386/repodata/repomd.xml: [Errno 14] PYCURL ERROR 6 - "Couldn't resolve host 'repos'"
Trying other mirror.
http://repos/repos/yum/releng/public/CentOS/6/noarch/repodata/repomd.xml: [Errno 14] PYCURL ERROR 6 - "Couldn't resolve host 'repos'"
Trying other mirror.
Package mpfr-2.4.1-6.el6.i686 already installed and latest version

now that 'repos' doesn't point to the last-used puppetmaster.  However, mock installs shouldn't be using only one host -- puppetmasters need to be able to go down without causing failures.  Mock should be using mirror lists as seen in the host's /etc/yum.repos.d/*.mirrors.

This is exacerbated by bug 938629, which is no longer adding 'repos' and 'puppet' to /etc/hosts.  It shouldn't have *removed* those entries though, except on [e-mail] where I did so manually.  So that's odd.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

This will need to be solved before the next TCW, as we need to reboot several of the puppet masters, and this bug means that clients are not resilient to failure of a single puppetmaster, as they should be.

Comment 2

[Chris AtLee [:catlee]]

WONTFIX in favor of your work in bug 1022763?