960571 - switch to https for build/test downloads and hg

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Chris AtLee [:catlee]]

We should change our test machinery to use https to download test/build files from ftp rather than plain http.

Ideally we can roll this out slowly so WebOps can monitor load as we do this.

We're hoping that we'll be able to have traffic between AWS and ftp.m.o go over the public internet in order to reduce load on the ipsec tunnel, which requires that we use https to guarantee file integrity.

Comment 1

[Ed Morley [:emorley]]

(In reply to Chris AtLee [:catlee] from comment #0)
> We're hoping that we'll be able to have traffic between AWS and ftp.m.o go
> over the public internet in order to reduce load on the ipsec tunnel

Presuming this will this help with bug 957502?

Comment 2

[Chris AtLee [:catlee]]

(In reply to Ed Morley [:edmorley UTC+0] from comment #1)
> (In reply to Chris AtLee [:catlee] from comment #0)
> > We're hoping that we'll be able to have traffic between AWS and ftp.m.o go
> > over the public internet in order to reduce load on the ipsec tunnel
> 
> Presuming this will this help with bug 957502?

that's the hope!

Comment 3

[Aki Sasaki (not active)]

Attachment: post_upload_https.diff

This will switch over all post_upload-based sendchanges/triggers over to https://ftp.m.o, which isn't "rolling out slowly".  Open to other ideas..?

I didn't touch the candidates url because it looks like stage.m.o doesn't have https enabled.

Comment 4

[Aki Sasaki (not active)]

Attachment: hgtool.diff

Stop clobbering hg share dirs on differences of http vs https.
Untested, but what could possibly go wrong?

Comment 5

[Aki Sasaki (not active)]

Scope creep!

Comment 6

[Aki Sasaki (not active)]

Comment on attachment 8366753 [details] [diff] [review]
post_upload_https.diff

[13:35]	<catlee>	aki: for bug 960571 you may want to wait to get the new hostname from bug 964486
[13:35]	<aki>	catlee: ok. we use that to explicitly change netflows?
[13:36]	<catlee>	yes

Comment 7

[Aki Sasaki (not active)]

Comment on attachment 8366753 [details] [diff] [review]
post_upload_https.diff

Er..

Comment 8

[Aki Sasaki (not active)]

Attachment: post_upload_https.diff

Comment 9

[Chris AtLee [:catlee]]

Landed https://hg.mozilla.org/build/mozharness/rev/8ac0f103b3de to switch traffic from http://ftp to https://ftp-ssl for now.

Comment 10

[Chris AtLee [:catlee]]

(In reply to Chris AtLee [:catlee] from comment #9)
> Landed https://hg.mozilla.org/build/mozharness/rev/8ac0f103b3de to switch
> traffic from http://ftp to https://ftp-ssl for now.

We still need public IPs for EC2, and routing table updates before this traffic will go over the public network.

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8366913 [details] [diff] [review]
post_upload_https.diff

I had wondered if we could set up the Apache config on http://ftp.m.o to redirect to https://ftp-ssl in some random way, but with a knob to control the proportion of the time it happens. Just for our machines preferably, which might get difficult when we're off the tunnel and the IP making the request isn't in 10.x.y.z any more. Could do something similar thing in mozharness, with much more direct control.

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Modifying the request I mean, pretty sure I saw a patch today that does that, just with some random() thrown in.

Comment 13

[Aki Sasaki (not active)]

http://mxr.mozilla.org/build/search?string=http%3A//ftp

Comment 14

[Chris AtLee [:catlee]]

Use https://hg-ssl.m.o for hg access if we don't think we can switch everything over to https://hg in a reasonable timeframe.

Comment 15

[Aki Sasaki (not active)]

Comment on attachment 8366764 [details] [diff] [review]
hgtool.diff

Hm, in a way we *don't* want this patch if we're switching to https.
We want all the http:// clones to go away.

Comment 16

[Aki Sasaki (not active)]

Attachment: b2g-inbound-https

It begins.

Comment 17

[Aki Sasaki (not active)]

Attachment: mozilla-inbound-talos

Comment 18

[Aki Sasaki (not active)]

Attachment: autoland

Comment 19

[Aki Sasaki (not active)]

braindump : http://hg.mozilla.org/build/braindump/rev/f84539717a09

Comment 20

[Aki Sasaki (not active)]

Attachment: buildbot-configs

Comment 21

[Aki Sasaki (not active)]

Comment on attachment 8368856 [details] [diff] [review]
autoland

https://hg.mozilla.org/build/autoland/rev/6d9e50722946

Comment 22

[Aki Sasaki (not active)]

Attachment: buildbotcustom

Comment 23

[Aki Sasaki (not active)]

Attachment: tools

Comment 24

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8368850 [details] [diff] [review]
mozilla-inbound-talos

Review of attachment 8368850 [details] [diff] [review]:
-----------------------------------------------------------------

I'm not overly familiar with this code, but I've taken a look at how it's used, and it seems like this should work fine.

Comment 25

[Aki Sasaki (not active)]

Attachment: tools

with 100% less hg-internal.

Comment 26

[Aki Sasaki (not active)]

Comment on attachment 8368896 [details] [diff] [review]
tools

https://hg.mozilla.org/build/tools/rev/935bb5461c1d

Comment 27

[Chris AtLee [:catlee]]

Comment on attachment 8368838 [details] [diff] [review]
b2g-inbound-https

Review of attachment 8368838 [details] [diff] [review]:
-----------------------------------------------------------------

and catlee saw that it was good

Comment 28

[Chris AtLee [:catlee]]

Comment on attachment 8368868 [details] [diff] [review]
buildbotcustom

Review of attachment 8368868 [details] [diff] [review]:
-----------------------------------------------------------------

do the tests work? I'm assuming twisted's getPage handles SSL ok

Comment 29

[Aki Sasaki (not active)]

Attachment: mozharness

http://hg.mozilla -> https://hg.mozilla.org
removal of hg-internal
http://ftp.mozilla.org -> https://ftp-ssl.mozilla.org

Comment 30

[Aki Sasaki (not active)]

Comment on attachment 8368838 [details] [diff] [review]
b2g-inbound-https

https://hg.mozilla.org/integration/b2g-inbound/rev/dfd275183653

Comment 31

[Aki Sasaki (not active)]

Comment on attachment 8366764 [details] [diff] [review]
hgtool.diff

Obsoleting due to comment 15.

Comment 32

[Rail Aliiev [:rail]]

Comment on attachment 8368866 [details] [diff] [review]
buildbot-configs

WCPGW? :)

Comment 33

[Aki Sasaki (not active)]

Comment on attachment 8368911 [details] [diff] [review]
mozharness

https://hg.mozilla.org/build/mozharness/rev/ce76df196bb1

Comment 34

[Aki Sasaki (not active)]

Comment on attachment 8368868 [details] [diff] [review]
buildbotcustom

https://hg.mozilla.org/build/buildbotcustom/rev/066534659fe8

Comment 35

[Aki Sasaki (not active)]

Comment on attachment 8368866 [details] [diff] [review]
buildbot-configs

https://hg.mozilla.org/build/buildbot-configs/rev/7720318925fd

Comment 36

[Aki Sasaki (not active)]

Attachment: puppet

Comment 37

[Aki Sasaki (not active)]

Attachment: partner-repacks

Comment 38

[Aki Sasaki (not active)]

Comment on attachment 8368850 [details] [diff] [review]
mozilla-inbound-talos

https://hg.mozilla.org/integration/mozilla-inbound/rev/1aa0a5f405fa

Comment 39

[Aki Sasaki (not active)]

Attachment: buildapi

Comment 40

[Aki Sasaki (not active)]

Attachment: cloud-tools

Comment 41

[Aki Sasaki (not active)]

Attachment: mozpool

Comment 42

[Aki Sasaki (not active)]

Comment on attachment 8368942 [details] [diff] [review]
buildapi

https://hg.mozilla.org/build/buildapi/rev/73dbae9cb3db

Comment 43

[Aki Sasaki (not active)]

Comment on attachment 8368951 [details] [diff] [review]
cloud-tools

https://hg.mozilla.org/build/cloud-tools/rev/432b3e0b983f

Comment 44

[Aki Sasaki (not active)]

Comment on attachment 8368838 [details] [diff] [review]
b2g-inbound-https

[Approval Request Comment]
Bug caused by (feature/regressing bug #): 
bug 957502

User impact if declined:
More tree closures due to infrastructure load on the tunnel.
We likely will not be able to redirect hg.mozilla.org traffic off the tunnel.
 
Testing completed (on m-c, etc.): 
Landed on b2g-inbound.

Risk to taking this patch (and alternatives if risky):
Could cause some build bustage, but we should catch it relatively quickly.

String or IDL/UUID changes made by this patch:
None.

Comment 45

[Aki Sasaki (not active)]

Comment on attachment 8368850 [details] [diff] [review]
mozilla-inbound-talos

[Approval Request Comment]
Bug caused by (feature/regressing bug #): 
bug 957502

User impact if declined:
More tree closures due to infrastructure load on the tunnel.
We likely will not be able to redirect hg.mozilla.org traffic off the tunnel.
 
Testing completed (on m-c, etc.): 
Landed on mozilla-inbound.

Risk to taking this patch (and alternatives if risky):
Could cause some build bustage, but we should catch it relatively quickly.

String or IDL/UUID changes made by this patch:
None.

Comment 46

[Aki Sasaki (not active)]

Attachment: buildbotcustom2

These are [hopefully all] the harder-to-find ones.

Comment 47

[Aki Sasaki (not active)]

Attachment: buildbot-configs2

Switch over to ftp-ssl and remove mirror urls.
This requires the buildbotcustom2 patch, or test-masters.sh dies on the mirror url removal.

Comment 48

[Aki Sasaki (not active)]

Comment on attachment 8368850 [details] [diff] [review]
mozilla-inbound-talos

We can probably use a=testing.

Comment 49

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/dfd275183653

Comment 50

[Aki Sasaki (not active)]

Comment on attachment 8368998 [details] [diff] [review]
buildbotcustom2

https://hg.mozilla.org/build/buildbotcustom/rev/5f7ccb6c3034

Comment 51

[Aki Sasaki (not active)]

Comment on attachment 8368999 [details] [diff] [review]
buildbot-configs2

https://hg.mozilla.org/build/cloud-tools/rev/432b3e0b983f

Comment 52

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] from comment #51)
> Comment on attachment 8368999 [details] [diff] [review]
> buildbot-configs2
> 
> https://hg.mozilla.org/build/cloud-tools/rev/432b3e0b983f

Er, https://hg.mozilla.org/build/buildbot-configs/rev/374f2d94035e

Comment 53

[Aki Sasaki (not active)]

Comment on attachment 8368934 [details] [diff] [review]
partner-repacks

https://hg.mozilla.org/build/partner-repacks/rev/1987c727dcec

Comment 54

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8368953 [details] [diff] [review]
mozpool

Review of attachment 8368953 [details] [diff] [review]:
-----------------------------------------------------------------

This is comments, docs, and a human-readable link in setup.py, but I've no problem with it.  It won't be necessary to ship a new version.

Comment 55

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/1aa0a5f405fa

Comment 56

[Aki Sasaki (not active)]

Comment on attachment 8368953 [details] [diff] [review]
mozpool

https://hg.mozilla.org/build/mozpool/rev/ca7e4096de0e

Comment 57

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 8368838 [details] [diff] [review]
b2g-inbound-https

let's get this in before merge.

Comment 58

[Aki Sasaki (not active)]

Uplifted to aurora:
https://hg.mozilla.org/releases/mozilla-aurora/rev/9737529dc0e2
https://hg.mozilla.org/releases/mozilla-aurora/rev/60803c5e3151

RyanVM has the other trees like a boss.

Comment 59

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-b2g26_v1_2/rev/4abbfef147cb
https://hg.mozilla.org/releases/mozilla-b2g26_v1_2/rev/1592ce49929c

https://hg.mozilla.org/releases/mozilla-b2g18/rev/6221852afb99
https://hg.mozilla.org/releases/mozilla-b2g18/rev/65371561a833

https://hg.mozilla.org/releases/mozilla-esr24/rev/fbfcffa000a6

I'll take care of the v1.1hd branch (Helix config) when I merge b2g18 there next.

Comment 60

[Aki Sasaki (not active)]

Comment on attachment 8368929 [details] [diff] [review]
puppet

https://hg.mozilla.org/build/puppet/rev/fc81396fae40

Comment 61

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-b2g18_v1_1_0_hd/rev/6221852afb99
https://hg.mozilla.org/releases/mozilla-b2g18_v1_1_0_hd/rev/65371561a833

With the Helix change included in the merge commit.

Comment 62

[Jordan Lund (:jlund)]

a mozharness patch has been merged into production :)

Comment 63

[Aki Sasaki (not active)]

Comment on attachment 8366913 [details] [diff] [review]
post_upload_https.diff

Sending        files/etc/post_upload.ini
Transmitting file data .
Committed revision 81800.

Comment 64

[Aki Sasaki (not active)]

http://hg.mozilla.org/build/mozharness/rev/eba30402acca landed to fix the wget ssl issue on 28.0b1.

I transplanted to default: http://hg.mozilla.org/build/mozharness/rev/618b1d93dca7
then backed it out on default: http://hg.mozilla.org/build/mozharness/rev/fd61adbd8703

Comment 65

[Aki Sasaki (not active)]

Merged mozharness; running a single locale nightly to test.

Comment 66

[Aki Sasaki (not active)]

make wget-en-US works ok without this patch, since bug 967452's patch landed.
Resolving this bug!

Comment 67

[Aki Sasaki (not active)]

Tbpl fix: https://hg.mozilla.org/webtools/tbpl/rev/2df551776fdf

Comment 68

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] from comment #67)
> Tbpl fix: https://hg.mozilla.org/webtools/tbpl/rev/2df551776fdf

IT rolled this out.

Hitting https://bugzilla.mozilla.org/show_bug.cgi?id=967452#c9 though; backing out post_upload.py

Comment 69

[Aki Sasaki (not active)]

Comment on attachment 8366913 [details] [diff] [review]
post_upload_https.diff

Backed out:
Sending        files/etc/post_upload.ini
Transmitting file data .
Committed revision 81811.

Comment 70

[Aki Sasaki (not active)]

OSX Jetpack also hit issues.  We need new wgets or smarter uses of wget.
https://bugzilla.mozilla.org/show_bug.cgi?id=967452#c10

Comment 71

[Aki Sasaki (not active)]

Reopening for post_upload.ini.

We need to fix wget on foopies and osx, at the least, before this can reland.

Comment 72

[Aki Sasaki (not active)]

Also windows wget for jetpack. Yay

Comment 73

[Aki Sasaki (not active)]

Attachment: fix_l10n

Comment 74

[Aki Sasaki (not active)]

Comment on attachment 8370971 [details] [diff] [review]
fix_l10n

https://hg.mozilla.org/build/buildbot-configs/rev/e7b3168ad5a2

Comment 75

[Jordan Lund (:jlund)]

https://hg.mozilla.org/build/buildbot-configs/rev/e7b3168ad5a2 -> is in production

Comment 76

[Ryan VanderMeulen [:RyanVM]]

(In reply to Aki Sasaki [:aki] from comment #67)
> Tbpl fix: https://hg.mozilla.org/webtools/tbpl/rev/2df551776fdf

In production :)

Comment 77

[Francesco Lodolo [:flod]]

My localized nightly just updated to 30, so we have now l10n central builds.

Comment 78

[[e-mail] (:bhearsum)]

(In reply to Francesco Lodolo [:flod] from comment #77)
> My localized nightly just updated to 30, so we have now l10n central builds.

Great! I assume you're on Mac, because the Windows nightly hasn't finished yet. Windows l10n builds should be getting updates again shortly after it does though.

Comment 79

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum [:bhearsum] from comment #78)
> (In reply to Francesco Lodolo [:flod] from comment #77)
> > My localized nightly just updated to 30, so we have now l10n central builds.
> 
> Great! I assume you're on Mac, because the Windows nightly hasn't finished
> yet. Windows l10n builds should be getting updates again shortly after it
> does though.

Windows seems to be working now too. Eg: https://aus4.mozilla.org/update/3/Firefox/14.0a1/20120222174716/WINNT_x86-msvc/de/nightly/default/default/default/update.xml

Comment 80

[Henrik Skupin [:whimboo][⌚️UTC+1]]

It would have been great if this change got more attention from other teams. I haven't seen any notification for it. As result our test automation for Mozmill was totally broken the whole last week given that pulsetranslator tried to grab the details via HTTP but not HTTPS. :(

I know that there are most likely dozen of tools involved here, which you might not all know. But especially because of that it would be kinda helpful to get information upfront, so that enough time exists to get tools updated. Can we make sure to do that in the future? Thanks.

Comment 81

[Aki Sasaki (not active)]

Sorry about that; it didn't even cross my mind.
However, it's not really clear what will and will not break external tools.  Is https the main thing you're concerned about when changed, or are there other things?

Comment 82

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Well, this time it was the HTTPS change. The update from Jgriffin for pulsetranslator made it work again. What I think could be helpful is to make an announcement in the future and cc the tools list, so people working on different tools for automation are aware of upcoming changes, which might break the current workflow.

Comment 83

[[e-mail] (:bhearsum)]

(In reply to Henrik Skupin (:whimboo) from comment #82)
> Well, this time it was the HTTPS change. The update from Jgriffin for
> pulsetranslator made it work again. What I think could be helpful is to make
> an announcement in the future and cc the tools list, so people working on
> different tools for automation are aware of upcoming changes, which might
> break the current workflow.

In this particular case we were working quickly to fix tree closing issues. There was a blog post made on the 5th though: http://atlee.ca/blog/posts/aws-networks-and-burning-trees.html

Comment 84

[Mark Reid [:mreid]]

This appears to have also broken Telemetry submissions since it bubbled up into the Telemetry Payload via the HISTOGRAMS_FILE_VERSION constant (which in turn comes from the "getSourceRepo" function in config/makefiles/rcs.mk)

It might be worth checking other uses of getSourceRepo to see if it's likely to cause any other problems, as well as possibly updating the comment in rcs.mk to indicate that the URI can be https.

Comment 85

[Aki Sasaki (not active)]

Unassigning, should someone want to take this bug while I'm out.
Aiui we need to land post_upload.py again once we deal with the blocking bugs.