493488 - Try server talos failing to download some mac builds

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Nick Thomas [:nthomas] (UTC+12)]

[nthomas@dm-wwwbuild01]$ find /builds/tryserver -type f -perm 600
/builds/tryserver/[e-mail]-bentest191/bentest191-macosx.dmg
/builds/tryserver/[e-mail]-private_493457/private_493457-macosx.dmg
/builds/tryserver/[e-mail]-try-3a38eb1d96c/try-3a38eb1d96c-macosx.dmg
/builds/tryserver/[e-mail]-try-456455311bf/try-456455311bf-macosx.dmg
/builds/tryserver/[e-mail]-mouse_and_key_queue/mouse_and_key_queue-macosx.dmg
/builds/tryserver/[e-mail]-currenttarget_opt/currenttarget_opt-macosx.dmg

Which leads to 
wget -nv -N --no-check-certificate http://build.mozilla.org/tryserver-builds/[e-mail]-private_493457/private_493457-macosx.dmg
WARNING: Certificate verification error for build.mozilla.org: self signed certificate in certificate chain
https://build.mozilla.org/tryserver-builds/[e-mail]-private_493457/private_493457-macosx.dmg:
12:33:25 ERROR 403: Forbidden.

Tracking down which macs are responsible.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Build: 1
Slave: mac3
File:  [e-mail]-try-3a38eb1d96c/try-3a38eb1d96c-macosx.dmg

Build: 3
Slave: mac9
File:  [e-mail]-bentest191/bentest191-macosx.dmg

Build: 4 
Slave: mac6
File:  [e-mail]-try-456455311bf/try-456455311bf-macosx.dmg

Build: 8
Slave: mac8
File:  [e-mail]-currenttarget_opt/currenttarget_opt-macosx.dmg

Build: 22
Slave: mac8
File:  [e-mail]-mouse_and_key_queue/mouse_and_key_queue-macosx.dmg

Build: 28
Slave: mac6
File:  [e-mail]-private_493457/private_493457-macosx.dmg

So spread over mac3/6/8/9, and all the slaves have umask=002 set. There's a mixture of push-to-try and patch upload, and m-1.9.1 and m-c. Also seems to be intermittent, eg builds 13, 15, 18, & 21 are also on mac3 and have world-readable dmg's; similarly build 6 on mac9.

Seems likely to be a regression from 486567.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

For completeness, packages that changed during the Thursday maintenance window on build.m.o:
                                      < kernel-PAE-2.6.18-128.1.1.el5.i686.rpm
                                      > kernel-PAE-2.6.18-128.1.10.el5.i686.rpm
lftp-3.7.8-1.el5.rf.i386.rpm          | lftp-3.7.12-1.el5.rf.i386.rpm
mercurial-1.1.2-1.el5.rf.i386.rpm     | mercurial-1.2.1-1.el5.rf.i386.rpm
nagios-nrpe-2.5.2-1.el5.rf.i386.rpm   | nagios-nrpe-2.12-1.el5.rf.i386.rpm
pango-1.14.9-3.el5.i386.rpm           | pango-1.14.9-5.el5_3.i386.rpm
php-5.1.6-23.el5.i386.rpm             | php-5.1.6-23.2.el5_3.i386.rpm
php-cli-5.1.6-23.el5.i386.rpm         | php-cli-5.1.6-23.2.el5_3.i386.rpm
php-common-5.1.6-23.el5.i386.rpm      | php-common-5.1.6-23.2.el5_3.i386.rpm
php-pdo-5.1.6-23.el5.i386.rpm         | php-pdo-5.1.6-23.2.el5_3.i386.rpm
rsync-3.0.5-1.el5.rf.i386.rpm         | rsync-3.0.6-1.el5.rf.i386.rpm
subversion-1.5.5-0.1.el5.rf.i386.rpm  | subversion-1.6.2-1.el5.rf.i386.rpm
syslinux-3.73-1.el5.rf.i386.rpm       | syslinux-3.80-1.el5.rf.i386.rpm
                                      > yum-utils-1.1.16-13.el5.noarch.rpm

(based on the contents of /var/log/rpmpkgs.1 and .2, dated May 16 and 9 respectively, modified diff --side-by-side output). Since we're getting this on push-to-try too we can probably rule out the php changes, and there's no openssl/openssh change which would affect uploading from the slaves.

Comment 3

[[e-mail] (:bhearsum)]

Hmm, I removed the chmod step because it looked like fixing the umask in the .tac file would be enough. I'll have a look here.

Comment 4

[[e-mail] (:bhearsum)]

(In reply to comment #1)
> So spread over mac3/6/8/9, and all the slaves have umask=002 set.

Looks like the umask was changed on 6 and 8 but the slave not restarted. I've done that there.

Looks like the failures from 3 and 9 happened before the slaves were restarted for the new umask. (Builds happened around 8am, restarts happened around 10am).

I think we're ok here now, I'll keep an eye on things today, though.

Comment 5

[[e-mail] (:bhearsum)]

We still seem to be hitting this.

Comment 6

[[e-mail] (:bhearsum)]

Ok, slave04, 06, and 08 still don't have the correct umask. The slaves are way too busy to fix it at the moment, I'll grab them when I can. In the meantime, I've got a script setup to chmod dmg files on build.m.o to 664 - hopefully this will fix things in the meantime.

Comment 7

[[e-mail] (:bhearsum)]

This should be all fixed now - really.