switch to https for build/test downloads and hg

RESOLVED FIXED

Status

defect
--
major
RESOLVED FIXED
5 years ago
a year ago

People

(Reporter: catlee, Unassigned)

Tracking

unspecified
x86_64
Linux
Dependency tree / graph

Firefox Tracking Flags

(firefox27 wontfix, firefox28 fixed, firefox29 fixed, firefox-esr24 fixed, b2g18 fixed, b2g-v1.1hd fixed, b2g-v1.2 fixed, b2g-v1.3 fixed, b2g-v1.4 fixed)

Details

(Whiteboard: [qa-])

Attachments

(16 attachments, 3 obsolete attachments)

1.33 KB, patch
nthomas
: review+
Details | Diff | Splinter Review
8.42 KB, patch
catlee
: review+
Details | Diff | Splinter Review
1.66 KB, patch
jgriffin
: review+
Details | Diff | Splinter Review
1.66 KB, patch
rail
: review+
Details | Diff | Splinter Review
27.69 KB, patch
rail
: review+
Details | Diff | Splinter Review
11.17 KB, patch
catlee
: review+
Details | Diff | Splinter Review
40.59 KB, patch
rail
: review+
Details | Diff | Splinter Review
135.93 KB, patch
rail
: review+
Details | Diff | Splinter Review
16.92 KB, patch
dustin
: review+
Details | Diff | Splinter Review
1.37 KB, patch
coop
: review+
Details | Diff | Splinter Review
7.39 KB, patch
catlee
: review+
Details | Diff | Splinter Review
5.01 KB, patch
rail
: review+
Details | Diff | Splinter Review
2.08 KB, patch
dustin
: review+
Details | Diff | Splinter Review
9.99 KB, patch
rail
: review+
Details | Diff | Splinter Review
20.69 KB, patch
rail
: review+
Details | Diff | Splinter Review
2.89 KB, patch
bhearsum
: review+
Details | Diff | Splinter Review
(Reporter)

Description

5 years ago
We should change our test machinery to use https to download test/build files from ftp rather than plain http.

Ideally we can roll this out slowly so WebOps can monitor load as we do this.

We're hoping that we'll be able to have traffic between AWS and ftp.m.o go over the public internet in order to reduce load on the ipsec tunnel, which requires that we use https to guarantee file integrity.
(In reply to Chris AtLee [:catlee] from comment #0)
> We're hoping that we'll be able to have traffic between AWS and ftp.m.o go
> over the public internet in order to reduce load on the ipsec tunnel

Presuming this will this help with bug 957502?
(Reporter)

Comment 2

5 years ago
(In reply to Ed Morley [:edmorley UTC+0] from comment #1)
> (In reply to Chris AtLee [:catlee] from comment #0)
> > We're hoping that we'll be able to have traffic between AWS and ftp.m.o go
> > over the public internet in order to reduce load on the ipsec tunnel
> 
> Presuming this will this help with bug 957502?

that's the hope!
Blocks: 957502

Comment 3

5 years ago
Posted patch post_upload_https.diff (obsolete) — Splinter Review
This will switch over all post_upload-based sendchanges/triggers over to https://ftp.m.o, which isn't "rolling out slowly".  Open to other ideas..?

I didn't touch the candidates url because it looks like stage.m.o doesn't have https enabled.
Attachment #8366753 - Flags: review?(nthomas)

Comment 4

5 years ago
Posted patch hgtool.diff (obsolete) — Splinter Review
Stop clobbering hg share dirs on differences of http vs https.
Untested, but what could possibly go wrong?
Attachment #8366764 - Flags: review?(nthomas)

Comment 5

5 years ago
Scope creep!
Summary: switch to https for build/test downloads → switch to https for build/test downloads and hg

Comment 6

5 years ago
Comment on attachment 8366753 [details] [diff] [review]
post_upload_https.diff

[13:35]	<catlee>	aki: for bug 960571 you may want to wait to get the new hostname from bug 964486
[13:35]	<aki>	catlee: ok. we use that to explicitly change netflows?
[13:36]	<catlee>	yes
Attachment #8366753 - Flags: review?(nthomas) → review-

Comment 7

5 years ago
Comment on attachment 8366753 [details] [diff] [review]
post_upload_https.diff

Er..
Attachment #8366753 - Attachment is obsolete: true
Attachment #8366753 - Flags: review-

Comment 8

5 years ago
Attachment #8366913 - Flags: review?(nthomas)
(Reporter)

Comment 10

5 years ago
(In reply to Chris AtLee [:catlee] from comment #9)
> Landed https://hg.mozilla.org/build/mozharness/rev/8ac0f103b3de to switch
> traffic from http://ftp to https://ftp-ssl for now.

We still need public IPs for EC2, and routing table updates before this traffic will go over the public network.
Attachment #8366764 - Flags: review?(nthomas) → review+
Comment on attachment 8366913 [details] [diff] [review]
post_upload_https.diff

I had wondered if we could set up the Apache config on http://ftp.m.o to redirect to https://ftp-ssl in some random way, but with a knob to control the proportion of the time it happens. Just for our machines preferably, which might get difficult when we're off the tunnel and the IP making the request isn't in 10.x.y.z any more. Could do something similar thing in mozharness, with much more direct control.
Attachment #8366913 - Flags: review?(nthomas) → review+
Modifying the request I mean, pretty sure I saw a patch today that does that, just with some random() thrown in.
(Reporter)

Comment 14

5 years ago
Use https://hg-ssl.m.o for hg access if we don't think we can switch everything over to https://hg in a reasonable timeframe.
Depends on: 965911
Comment on attachment 8366764 [details] [diff] [review]
hgtool.diff

Hm, in a way we *don't* want this patch if we're switching to https.
We want all the http:// clones to go away.
It begins.
Assignee: nobody → aki
Attachment #8368838 - Flags: review?(catlee)
Attachment #8368850 - Flags: review?(jmaher)

Updated

5 years ago
Attachment #8368850 - Flags: review?(jgriffin)
Posted patch autolandSplinter Review
Attachment #8368856 - Flags: review?(rail)

Updated

5 years ago
Attachment #8368856 - Flags: review?(rail) → review+
Attachment #8368866 - Flags: review?(bugspam.Callek)
Attachment #8368868 - Flags: review?(catlee)
Posted patch tools (obsolete) — Splinter Review
Attachment #8368881 - Flags: review?(rail)
Comment on attachment 8368850 [details] [diff] [review]
mozilla-inbound-talos

Review of attachment 8368850 [details] [diff] [review]:
-----------------------------------------------------------------

I'm not overly familiar with this code, but I've taken a look at how it's used, and it seems like this should work fine.
Attachment #8368850 - Flags: review?(jgriffin) → review+

Updated

5 years ago
Attachment #8368850 - Flags: review?(jmaher)
Posted patch toolsSplinter Review
with 100% less hg-internal.
Attachment #8368881 - Attachment is obsolete: true
Attachment #8368881 - Flags: review?(rail)
Attachment #8368896 - Flags: review?(rail)

Updated

5 years ago
Attachment #8368896 - Flags: review?(rail) → review+
(Reporter)

Comment 27

5 years ago
Comment on attachment 8368838 [details] [diff] [review]
b2g-inbound-https

Review of attachment 8368838 [details] [diff] [review]:
-----------------------------------------------------------------

and catlee saw that it was good
Attachment #8368838 - Flags: review?(catlee) → review+
(Reporter)

Comment 28

5 years ago
Comment on attachment 8368868 [details] [diff] [review]
buildbotcustom

Review of attachment 8368868 [details] [diff] [review]:
-----------------------------------------------------------------

do the tests work? I'm assuming twisted's getPage handles SSL ok
Attachment #8368868 - Flags: review?(catlee) → review+
Comment on attachment 8366764 [details] [diff] [review]
hgtool.diff

Obsoleting due to comment 15.
Attachment #8366764 - Attachment is obsolete: true

Updated

5 years ago
Attachment #8368911 - Flags: review?(rail) → review+
Comment on attachment 8368866 [details] [diff] [review]
buildbot-configs

WCPGW? :)
Attachment #8368866 - Flags: review?(bugspam.Callek) → review+
Posted patch puppetSplinter Review
Attachment #8368929 - Flags: review?(dustin)
Attachment #8368934 - Flags: review?(coop)
Posted patch buildapiSplinter Review
Attachment #8368942 - Flags: review?(catlee)
Posted patch cloud-toolsSplinter Review
Attachment #8368951 - Flags: review?(rail)
Posted patch mozpoolSplinter Review
Attachment #8368953 - Flags: review?(dustin)

Updated

5 years ago
Attachment #8368951 - Flags: review?(rail) → review+
(Reporter)

Updated

5 years ago
Attachment #8368942 - Flags: review?(catlee) → review+
Comment on attachment 8368838 [details] [diff] [review]
b2g-inbound-https

[Approval Request Comment]
Bug caused by (feature/regressing bug #): 
bug 957502

User impact if declined:
More tree closures due to infrastructure load on the tunnel.
We likely will not be able to redirect hg.mozilla.org traffic off the tunnel.
 
Testing completed (on m-c, etc.): 
Landed on b2g-inbound.

Risk to taking this patch (and alternatives if risky):
Could cause some build bustage, but we should catch it relatively quickly.

String or IDL/UUID changes made by this patch:
None.
Attachment #8368838 - Flags: approval-mozilla-b2g28?
Attachment #8368838 - Flags: approval-mozilla-b2g26?
Attachment #8368838 - Flags: approval-mozilla-b2g18?
Attachment #8368838 - Flags: approval-mozilla-aurora?
Comment on attachment 8368850 [details] [diff] [review]
mozilla-inbound-talos

[Approval Request Comment]
Bug caused by (feature/regressing bug #): 
bug 957502

User impact if declined:
More tree closures due to infrastructure load on the tunnel.
We likely will not be able to redirect hg.mozilla.org traffic off the tunnel.
 
Testing completed (on m-c, etc.): 
Landed on mozilla-inbound.

Risk to taking this patch (and alternatives if risky):
Could cause some build bustage, but we should catch it relatively quickly.

String or IDL/UUID changes made by this patch:
None.
Attachment #8368850 - Flags: approval-mozilla-release?
Attachment #8368850 - Flags: approval-mozilla-esr24?
Attachment #8368850 - Flags: approval-mozilla-beta?
Attachment #8368850 - Flags: approval-mozilla-b2g28?
Attachment #8368850 - Flags: approval-mozilla-b2g26?
Attachment #8368850 - Flags: approval-mozilla-b2g18?
Attachment #8368850 - Flags: approval-mozilla-aurora?
These are [hopefully all] the harder-to-find ones.
Attachment #8368998 - Flags: review?(rail)
Switch over to ftp-ssl and remove mirror urls.
This requires the buildbotcustom2 patch, or test-masters.sh dies on the mirror url removal.
Attachment #8368999 - Flags: review?(rail)
Comment on attachment 8368850 [details] [diff] [review]
mozilla-inbound-talos

We can probably use a=testing.
Attachment #8368850 - Flags: approval-mozilla-release?
Attachment #8368850 - Flags: approval-mozilla-esr24?
Attachment #8368850 - Flags: approval-mozilla-beta?
Attachment #8368850 - Flags: approval-mozilla-b2g28?
Attachment #8368850 - Flags: approval-mozilla-b2g26?
Attachment #8368850 - Flags: approval-mozilla-b2g18?
Attachment #8368850 - Flags: approval-mozilla-aurora?

Updated

5 years ago
Attachment #8368998 - Flags: review?(rail) → review+

Updated

5 years ago
Attachment #8368999 - Flags: review?(rail) → review+
Comment hidden (typo)
Attachment #8368934 - Flags: review?(coop) → review+
Attachment #8368929 - Flags: review?(dustin) → review+
Comment on attachment 8368953 [details] [diff] [review]
mozpool

Review of attachment 8368953 [details] [diff] [review]:
-----------------------------------------------------------------

This is comments, docs, and a human-readable link in setup.py, but I've no problem with it.  It won't be necessary to ship a new version.
Attachment #8368953 - Flags: review?(dustin) → review+
https://hg.mozilla.org/mozilla-central/rev/1aa0a5f405fa
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED

Updated

5 years ago
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Whiteboard: [leave open]
Comment on attachment 8368838 [details] [diff] [review]
b2g-inbound-https

let's get this in before merge.
Attachment #8368838 - Flags: approval-mozilla-b2g28?
Attachment #8368838 - Flags: approval-mozilla-b2g28+
Attachment #8368838 - Flags: approval-mozilla-b2g26?
Attachment #8368838 - Flags: approval-mozilla-b2g26+
Attachment #8368838 - Flags: approval-mozilla-b2g18?
Attachment #8368838 - Flags: approval-mozilla-b2g18+
Attachment #8368838 - Flags: approval-mozilla-aurora?
Attachment #8368838 - Flags: approval-mozilla-aurora+
a mozharness patch has been merged into production :)
Comment on attachment 8366913 [details] [diff] [review]
post_upload_https.diff

Sending        files/etc/post_upload.ini
Transmitting file data .
Committed revision 81800.
Attachment #8366913 - Flags: checked-in+

Updated

5 years ago
Depends on: 967452
Merged mozharness; running a single locale nightly to test.
make wget-en-US works ok without this patch, since bug 967452's patch landed.
Resolving this bug!
Status: REOPENED → RESOLVED
Last Resolved: 5 years ago5 years ago
Resolution: --- → FIXED
(In reply to Aki Sasaki [:aki] from comment #67)
> Tbpl fix: https://hg.mozilla.org/webtools/tbpl/rev/2df551776fdf

IT rolled this out.

Hitting https://bugzilla.mozilla.org/show_bug.cgi?id=967452#c9 though; backing out post_upload.py
Comment on attachment 8366913 [details] [diff] [review]
post_upload_https.diff

Backed out:
Sending        files/etc/post_upload.ini
Transmitting file data .
Committed revision 81811.
Attachment #8366913 - Flags: checked-in+ → checked-in-
OSX Jetpack also hit issues.  We need new wgets or smarter uses of wget.
https://bugzilla.mozilla.org/show_bug.cgi?id=967452#c10
Reopening for post_upload.ini.

We need to fix wget on foopies and osx, at the least, before this can reland.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Whiteboard: [leave open]
Also windows wget for jetpack. Yay
Whiteboard: [qa-]
Posted patch fix_l10nSplinter Review
Attachment #8370971 - Flags: review?(bhearsum)
Attachment #8370971 - Flags: review?(bhearsum) → review+
Depends on: 968433
(In reply to Aki Sasaki [:aki] from comment #67)
> Tbpl fix: https://hg.mozilla.org/webtools/tbpl/rev/2df551776fdf

In production :)
My localized nightly just updated to 30, so we have now l10n central builds.
(In reply to Francesco Lodolo [:flod] from comment #77)
> My localized nightly just updated to 30, so we have now l10n central builds.

Great! I assume you're on Mac, because the Windows nightly hasn't finished yet. Windows l10n builds should be getting updates again shortly after it does though.
(In reply to Ben Hearsum [:bhearsum] from comment #78)
> (In reply to Francesco Lodolo [:flod] from comment #77)
> > My localized nightly just updated to 30, so we have now l10n central builds.
> 
> Great! I assume you're on Mac, because the Windows nightly hasn't finished
> yet. Windows l10n builds should be getting updates again shortly after it
> does though.

Windows seems to be working now too. Eg: https://aus4.mozilla.org/update/3/Firefox/14.0a1/20120222174716/WINNT_x86-msvc/de/nightly/default/default/default/update.xml
It would have been great if this change got more attention from other teams. I haven't seen any notification for it. As result our test automation for Mozmill was totally broken the whole last week given that pulsetranslator tried to grab the details via HTTP but not HTTPS. :(

I know that there are most likely dozen of tools involved here, which you might not all know. But especially because of that it would be kinda helpful to get information upfront, so that enough time exists to get tools updated. Can we make sure to do that in the future? Thanks.
Depends on: 968169
Sorry about that; it didn't even cross my mind.
However, it's not really clear what will and will not break external tools.  Is https the main thing you're concerned about when changed, or are there other things?
Well, this time it was the HTTPS change. The update from Jgriffin for pulsetranslator made it work again. What I think could be helpful is to make an announcement in the future and cc the tools list, so people working on different tools for automation are aware of upcoming changes, which might break the current workflow.
(In reply to Henrik Skupin (:whimboo) from comment #82)
> Well, this time it was the HTTPS change. The update from Jgriffin for
> pulsetranslator made it work again. What I think could be helpful is to make
> an announcement in the future and cc the tools list, so people working on
> different tools for automation are aware of upcoming changes, which might
> break the current workflow.

In this particular case we were working quickly to fix tree closing issues. There was a blog post made on the 5th though: http://atlee.ca/blog/posts/aws-networks-and-burning-trees.html

Updated

5 years ago
Depends on: 971155

Updated

5 years ago
Depends on: 971157

Updated

5 years ago
Depends on: 971160
(Reporter)

Updated

5 years ago
Blocks: 971846
This appears to have also broken Telemetry submissions since it bubbled up into the Telemetry Payload via the HISTOGRAMS_FILE_VERSION constant (which in turn comes from the "getSourceRepo" function in config/makefiles/rcs.mk)

It might be worth checking other uses of getSourceRepo to see if it's likely to cause any other problems, as well as possibly updating the comment in rcs.mk to indicate that the URI can be https.
Unassigning, should someone want to take this bug while I'm out.
Aiui we need to land post_upload.py again once we deal with the blocking bugs.
Assignee: aki → nobody
(Reporter)

Updated

2 years ago
Status: REOPENED → RESOLVED
Last Resolved: 5 years ago2 years ago
Resolution: --- → FIXED
Component: General Automation → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.