637577 - test and talos jobs are not running on linux64 slaves

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P1)

Description

[Timothy Nikkel (:tnikkel)]

Affecting at least mozilla-central and try.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Looks like talos-r3-fed64-* started backing between 4 and 5pm Pacific today.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Fallout from 
http://hg.mozilla.org/build/puppet-manifests/rev/829ee80177dc#l5.1

Feb 28 16:51:37 talos-r3-fed64-003 puppetd[1804]: Starting catalog run
Feb 28 16:51:41 talos-r3-fed64-003 puppetd[1804]: (//Node[talos-r3-fed64-003.build.mozilla.org]/talosslave/talos_fedora/Package[python-devel]/ensure) change from absent to latest failed: Could not update:  at /etc/puppet/manifests/os/talos_fedora.pp:27

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

So those machines just sit there polling puppet every few minutes. 

AFAICT, fedora12-x86_64/test/RPMs/python-devel-2.6.2-2.fc12.x86_64.rpm wasn't copied from staging/ to production/:

[root@production-puppet N]# find /N -name '*python-devel*'
/N/staging/fedora12-x86_64/test/RPMs/python-devel-2.6.2-2.fc12.x86_64.rpm
[root@production-puppet N]#

[root@scl-production-puppet production]# find /builds/production/ -name '*python-devel*'
[root@scl-production-puppet production]#

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

I copied python-devel-2.6.2-2.fc12.x86_64.rpm from staging to production at mpt-p-p, and from there to mv-p-p and scl-p-p. My test slave picked that up and proceeded to start buildbot and collect a job.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Pending job count is collapsing as we clear the backlog. For m-c and tracemonkey and m-c only the most recent test request for each builder (tinderbox column) will get done. If there are regressions since 16:30 today we can trigger jobs to bisect the changes, where we = rail. ;-)