738775 - macosx fuzzing jobs are failing fast with an error downloading the build

Status: VERIFIED FIXED

(Release Engineering :: General, defect)

Description

[Mike Taylor [:bear]]

I've seen 3 darwin9 slaves all with this failure:

Looking in https://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx-debug/...
Traceback (most recent call last):
  File "fuzzing/dom/automation/bot.py", line 203, in <module>
    buildUsed = downloadBuild.downloadLatestBuild(buildType)
  File "/builds/slave/fuzzer-macosx/fuzzing/dom/automation/downloadBuild.py", line 136, in downloadLatestBuild
    raise Exception("No builds in " + buildsDir + "!")
Exception: No builds in https://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx-debug/!

and when I look at that ftp url I see builds inside.

Comment 1

[Jesse Ruderman]

Hmm, seems to work locally. Could it be that the slaves have a broken curl, or are unable to access https://ftp.mozilla.org/?

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Broken curl:

moz2-darwin9-slave23:~ cltbld$ curl https://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx-debug/
curl: (60) SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). The default
 bundle is named curl-ca-bundle.crt; you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

Works on darwin10. WONTFIX given these machines are going to go away, or morph to move the fuzzing jobs around ?

Comment 3

[Jesse Ruderman]

Fun.  When are the machines going away?

As long as this isn't interfering with the Mac OS X 10.6+ machines getting jobs, it doesn't really bother me.

Comment 4

[Chris Cooper [:coop] (he/him)]

(In reply to Nick Thomas [:nthomas] from comment #2) 
> curl performs SSL certificate verification by default, using a "bundle"
>  of Certificate Authority (CA) public keys (CA certs). The default
>  bundle is named curl-ca-bundle.crt; you can specify an alternate file
>  using the --cacert option.
> If this HTTPS server uses a certificate signed by a CA represented in
>  the bundle, the certificate verification probably failed due to a
>  problem with the certificate (it might be expired, or the name might
>  not match the domain name in the URL).
> If you'd like to turn off curl's verification of the certificate, use
>  the -k (or --insecure) option.

Can we not just add the -k flag on darwin9

Comment 6

[Armen [:armenzg]]

I would like to drop darwin9 support if possible. IT would not be happy if we try to maintain these machines as production machines.

Comment 7

[Chris Cooper [:coop] (he/him)]

(In reply to Armen Zambrano G. [:armenzg] - Release Engineer from comment #6)
> I would like to drop darwin9 support if possible. IT would not be happy if
> we try to maintain these machines as production machines.

We need some small pool of these machines around for the foreseeable future due to the ESR branch, so they might as well still be fuzzing when not building ESR (~once every 6 weeks).

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Fair call. The fuzzer could just switch to wget, since we already install the equivalent cert bundle for that using puppet. eg a random file request using wget:

moz2-darwin9-slave23:~ cltbld$ wget https://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx-debug/1330821607/firefox-13.0a1.en-US.mac.checksums
--2012-03-26 14:53:02--  https://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx-debug/1330821607/firefox-13.0a1.en-US.mac.checksums
Resolving ftp.mozilla.org... 63.245.209.137
Connecting to ftp.mozilla.org|63.245.209.137|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2055 (2.0K) [application/octet-stream]
Saving to: `firefox-13.0a1.en-US.mac.checksums'

100%[=====================================================================================================================================================================================================>] 2,055       --.-K/s   in 0s     

2012-03-26 14:53:02 (24.5 MB/s) - `firefox-13.0a1.en-US.mac.checksums' saved [2055/2055]

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Spot check says that works on moz2-darwin10-slaveNN, moz2-linux-slaveNN, linux-ix-slaveNN, moz2-linux64-slaveNN, linux64-ix-slaveNN, and w32-ix-slaveNN too. 

Jesse owns the fuzzer script, over to you ?

Comment 10

[Jesse Ruderman]

I changed it to use wget on Mac. (It was already using wget on non-Mac.) Hopefully this fixes the problem :)

Comment 11

[Mike Taylor [:bear]]

I'm seeing green fuzzing jobs on darwin9 - thanks Jesse!