wget on mac builders refuses to download from https://s3-us-west-2.amazonaws.com

RESOLVED FIXED

Status

Release Engineering
Mozharness
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: rail, Assigned: Callek)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

20:10:23     INFO -  (cd /builds/slave/rel-date_fx_m64_l10n_rpk-00000/build/date/obj-l10n/dist/ && wget --no-cache -nv -N  'https://queue.taskcluster.net/v1/task/Zcg23IJyT7KfU8X1q6ClVw/artifacts/public/build/firefox-42.0.en-US.mac.dmg')
20:10:25     INFO -  ERROR: certificate common name `*.s3-us-west-2.amazonaws.com' doesn't match requested host name `s3-us-west-2.amazonaws.com'.
20:10:25     INFO -  To connect to s3-us-west-2.amazonaws.com insecurely, use `--no-check-certificate'.
20:10:25     INFO -  make: *** [wget-en-US] Error 5

It does work on linux, will check windows.
+windows
Summary: wget on mac builders refuses to download from https://s3-us-west-2.amazonaws.com → wget on mac and windows builders refuses to download from https://s3-us-west-2.amazonaws.com
This is what we have on windows (wget 1.10.2):

C:\Users\cltbld>wget https://queue.taskcluster.net/v1/task/Zcg23IJyT7KfU8X1q6ClVw/artifacts/public/build/firefox-42.0.en-US.mac.dmg
--07:44:10--  https://queue.taskcluster.net/v1/task/Zcg23IJyT7KfU8X1q6ClVw/artifacts/public/build/firefox-42.0.en-US.mac.dmg
           => `firefox-42.0.en-US.mac.dmg'
Resolving queue.taskcluster.net... 54.197.225.245, 23.23.110.46, 54.243.78.74
Connecting to queue.taskcluster.net|54.197.225.245|:443... connected.
ERROR: Certificate verification error for queue.taskcluster.net: certificate sig
nature failure
To connect to queue.taskcluster.net insecurely, use `--no-check-certificate'.
Unable to establish SSL connection.

C:\Users\cltbld>

As a temporary work around I pushed https://hg.mozilla.org/projects/date/rev/1546017f6d36#l2.13
Mac uses wget 1.12-1, Linux is 1.15-2.el6, widows is 1.10.2 (from mozilla-build).

Looks like mac can recognize the queue.t.n's cert, but can't handle the S3 situation. I suspect this is related to SNI, which is not supported in 1.12 IIRC.

Windows also needs a fresher ca-bundle.crt to be able to talk to queue.t.n.
Dustin, how problematic would be to upgrade wget on Mac and Windows? I believe, Mac will be relatively easy, but Windows may need some love, because we may need to deploy wget in parallel with the coming with MozillaBuild 1.9.0.
Flags: needinfo?(dustin)
Yes, it should be relatively easy on OS X.

Probably on Windows the best solution is to (temporarily) use --no-check-certificate, just to avoid adding one more moving target to the current Windows deployment work.
Flags: needinfo?(dustin)
(In reply to Dustin J. Mitchell [:dustin] from comment #5)
> Probably on Windows the best solution is to (temporarily) use
> --no-check-certificate, just to avoid adding one more moving target to the
> current Windows deployment work.

This is my current work around on date, but I'm not sure if https://hg.mozilla.org/projects/date/rev/1546017f6d36#l2.13 passes a review :)
I wonder if we should use tooltool here instead of deploying the package system wide...
Tooltool's not a bad idea.  We could do the install from tooltool into the windows images for taskcluster later.
I just filed bug 1214815 for serving public artifacts using cloudfront for people outside EC2.
Inside EC2 we rely on s3-copy-proxy, so that'll still have SNI, this is a cost measure.

But cloudfront supports dedicated IPs for SSL (no SNI), at $600/month per certificate, so that's
an easy switch to flip. I'll teach queue.tc.net to redirect to CF once DNS setup and propagated.

Later we might do cloudfront for private artifacts too (bug 1214818), assuming it's faster than S3
for people in Europe who have been complaining about download speed. If it's not faster we might drop
using cloudfront, supporting non-SNI clients is more of a side-effect than a feature :)
So we are now serving public artifacts from cloudfront, assuming you're not running in EC2 in which case
you'll be redirected to an S3 bucket near you.

Anyways, the CF distribution is configured to support legacy clients that don't have SNI.
So maybe you don't need to work around for now, long-term we should upgrade clients.
Oh, cool. Is this something new? I had these issues just yesterday.

Upgrading wget would be great though.
(Assignee)

Updated

2 years ago
Assignee: nobody → bugspam.Callek
(Assignee)

Comment 12

2 years ago
Created attachment 8709200 [details] [diff] [review]
[puppet] osx

I created this patch, with OSX 10.6, 10.7 and 10.10 in mind!

If we have any other OSX versions in production, please tell me their hostnames and/or who can help me build for them and I'll do so. Otherwise we can just mark it as "not for those versions" in puppet.

This was tested for those 3 versions, and seems to work fine!
Attachment #8709200 - Flags: review?(rail)
Comment on attachment 8709200 [details] [diff] [review]
[puppet] osx

(In reply to Justin Wood (:Callek) from comment #12) 
> This was tested for those 3 versions, and seems to work fine!

Wheee! Thank you for this fix!
Attachment #8709200 - Flags: review?(rail) → review+
(Assignee)

Comment 14

2 years ago
Hey Rail,

I'm off tomorrow (tuesday) so if you want to land it for me and watch for puppet errors, and/or sheriffs complaining about any brokenness, I'm game for that. Otherwise I'll just land on Wednesday. Clear the n-i if you land, or I will when I do.
Flags: needinfo?(rail)
Depends on: 1054812
(Assignee)

Updated

2 years ago
No longer depends on: 1054812
(Assignee)

Comment 16

2 years ago
changing this bug to reflect *just* OSX since we already had a bug about windows in Bug 1054812, which I marked as also blocking relpromo
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
Summary: wget on mac and windows builders refuses to download from https://s3-us-west-2.amazonaws.com → wget on mac builders refuses to download from https://s3-us-west-2.amazonaws.com
(Assignee)

Updated

2 years ago
Duplicate of this bug: 971155
You need to log in before you can comment on or make changes to this bug.