590029 - 403 error when some machines try to hit stage-clobberer

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[John Ford [:jhford] CET/CEST Berlin Time]

I just recently had a bunch of failing tag runs due to clobberer not working.

running:
python tag/tools/clobberer/clobberer.py -s tools http://build.mozilla.org/stage-clobberer/index.php nothing tag tag mv-moz2-linux-ix-slave01 http://staging-master.build.mozilla.org:8011/

results in:
Checking clobber URL: http://build.mozilla.org/stage-clobberer/index.php?master=http%3A%2F%2Fstaging-master.build.mozilla.org%3A8011%2F&slave=mv-moz2-linux-ix-slave01&builddir=tag&branch=nothing&buildername=tag
Error contacting server
program finished with exit code 1

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

It's returning 'HTTP/1.1 403 Forbidden' after redirecting from http to https. Which is strange, there's 
 Allow from 10.250.48.0/255.255.254.0
in the config, which should include 10.250.49.199 (mv-moz2-linux-ix-slave01). We have the same Allow lines for both staging and production.

Can't see the apache logs since the kernel upgrades chmod'd /var/log/httpd/.

Comment 2

[[e-mail] (:bhearsum)]

I think this belongs in Server Ops based on Nick's comment:
(In reply to comment #1)
> It's returning 'HTTP/1.1 403 Forbidden' after redirecting from http to https.
> Which is strange, there's 
>  Allow from 10.250.48.0/255.255.254.0
> in the config, which should include 10.250.49.199 (mv-moz2-linux-ix-slave01).
> We have the same Allow lines for both staging and production.
> 
> Can't see the apache logs since the kernel upgrades chmod'd /var/log/httpd/.

Comment 3

[Shyam Mani [:fox2mike]]

Err, so I can see the logs :

[root@dm-wwwbuild01 ~]# su - nthomas

[nthomas@dm-wwwbuild01 ~]$ cd /var/log/httpd/build.mozilla.org/

[nthomas@dm-wwwbuild01 build.mozilla.org]$ tailf access_2010-08-25-12 

10.250.49.165 - - [25/Aug/2010:05:32:06 -0700] "GET /clobberer/index.php?master=http%253A%252F%252Fproduction-master02.build.mozilla.org%253A8010%252F&slave=mv-moz2-linux-ix-slave17&builddir=mobile-1.9.2-maemo5-gtk-l10n-nightly&branch=mozilla-1.9.2&buildername=Maemo+5+GTK+mozilla-1.9.2+l10n HTTP/1.1" 200 2848 "-" "Python-urllib/2.5"
10.2.71.250 - - [25/Aug/2010:05:32:07 -0700] "GET /clobberer/index.php?master=http%253A%252F%252Fproduction-master03.build.mozilla.org%253A8010%252F&slave=moz2-linux-slave16&builddir=mozilla-1.9.2-linux-l10n-nightly&branch=mozilla-1.9.2&buildername=Firefox+mozilla-1.9.2+linux+l10n+nightly HTTP/1.1" 200 2481 "-" "Python-urllib/2.5"
10.250.49.159 - - [25/Aug/2010:05:32:09 -0700] "GET /clobberer/index.php?master=http%253A%252F%252Fproduction-master03.build.mozilla.org%253A8010%252F&slave=mv-moz2-linux-ix-slave11&builddir=maple-mobile-browser-maemo5-qt-build&branch=maple&buildername=Maemo+5+QT+maple+build HTTP/1.1" 200 3388 "-" "Python-urllib/2.5"
10.2.71.220 - - [25/Aug/2010:05:33:09 -0700] "GET /clobberer/index.php?master=http%253A%252F%252Fproduction-master03.build.mozilla.org%253A8010%252F&slave=moz2-linux-slave19&builddir=mozilla-1.9.2-linux-l10n-nightly&branch=mozilla-1.9.2&buildername=Firefox+mozilla-1.9.2+linux+l10n+nightly HTTP/1.1" 200 1995 "-" "Python-urllib/2.5"
10.250.49.169 - - [25/Aug/2010:05:33:10 -0700] "GET /clobberer/index.php?master=http%253A%252F%252Fproduction-master02.build.mozilla.org%253A8010%252F&slave=mv-moz2-linux-ix-slave21&builddir=mobile-1.9.2-maemo5-gtk-l10n-nightly&branch=mozilla-1.9.2&buildername=Maemo+5+GTK+mozilla-1.9.2+l10n HTTP/1.1" 200 3948 "-" "Python-urllib/2.5"

is that the issue? Or you wanted us to look at why you're getting 403s?

Comment 4

[[e-mail] (:bhearsum)]

Not sure if the last comment is clear, updating the summary.

When trying to access http://build.mozilla.org/stage-clobberer/index.php?master=http%3A%2F%2Fstaging-master.build.mozilla.org%3A8011%2F&slave=mv-moz2-linux-ix-slave01&builddir=tag&branch=nothing&buildername=tag from, eg, mv-moz2-linux-ix-slave01, I get this (copied and pasted from links):

                                                                                                                            403 Forbidden 
                                                                Forbidden                                                                 
                                                                                                                                          
   You don't have permission to access /stage-clobberer/index.php on this server.                                                         
                                                                                                                                          
   ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────   
                                                                                                                                          
    Apache/2.2.3 (Red Hat) Server at build.mozilla.org Port 443

Comment 5

[Shyam Mani [:fox2mike]]

Appended an allow line, should be okay now.

Comment 6

[[e-mail] (:bhearsum)]

Thanks for looking at this Shyam.

Turns out the issue was that index.php was a symlink to another file, and we're not following symlinks. I've now removed the symlink and updated index.php to the latest clobberer

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

The symlink was my mistake. I was trying to add some sort of versioning, since we operate on a copy of the repo file rather than a repo itself. I reverted the symlink in production but forgot about staging. Sorry!