815124 - Switch buildbot wgets from ftp.mozilla.org to http://ftp-scl3.mozilla.com for bug 811063

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Ed Morley [:emorley]]

(In reply to Peter Radcliffe [:pir] from bug 811063 comment #51)
> I think we have plenty of examples now, thanks.
> 
> The issue is debugging. The nameservers get far too many requests to log
> absolutely everything and it's hard to work out if a handful of requests in
> a few million gets to the nameserver or if there's a network issue, the
> process is not responding or one of a dozen other reasons these requests
> aren't getting responded to.
> 
> We'll need to work through some possibilities. Easiest to test is taking
> dynect out of the path. I've added ftp-scl3.mozilla.com as a direct ip, not
> a CNAME, can you switch these wget downloads to paths under
> http://ftp-scl3.mozilla.com/pub/mozilla.org/ instead of ftp.mozilla.org,
> please?

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Blocked on ftp-scl3.mozilla.com being resolvable inside the build network.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [svn] Swap to ftp-scl3.m.c

Aki, this changes the output post_upload.py returns when we call make_upload, so we get the different domain in the properties like packageUrl, which then used to sendchange for tests. I've verified that test machines in scl1/scl3/mtv1 can all retrieve files using ftp-scl3.m.c.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 686313 [details] [diff] [review]
[svn] Swap to ftp-scl3.m.c

Committed revision 53458.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

This went live at 16:18, any compile jobs uploading after this will use ftp-scl3.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 686313 [details] [diff] [review]
[svn] Swap to ftp-scl3.m.c

Reverted:
Committed revision 53470.

TBPL gives errors like this:
Log file http://ftp-scl3.mozilla.com/pub/mozilla.org/firefox/try-builds/[e-mail]-786b11645825/try-macosx64-debug/try_mountainlion-debug_test-mochitest-2-bm37-tests1-macosx-build289.txt.gz not hosted on ftp.mozilla.org!
due to http://hg.mozilla.org/users/mstange_themasta.com/tinderboxpushlog/file/default/php/inc/RawGzLogDownloader.php#l18

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

digi reverted this by hand at 17:14. Any jobs which finished in the 60 mins before that will cause an error on TBPL. I'll get TBPL updated before trying to deploy this again.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 686313 [details] [diff] [review]
[svn] Swap to ftp-scl3.m.c

Landed again:
Committed revision 53520.

Will go live between 1230 and 1300 Pacific when puppet runs.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Went live at 12:59.

Comment 9

[Ed Morley [:emorley]]

Looking good, thank you :-)

Comment 10

[Ed Morley [:emorley]]

Per bug 811063 comment 81, could we try reverting this please? :-)

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

If this is wanted before I get back early January then please contact #build on irc. 

RelEng: Deployment docs at https://wiki.mozilla.org/ReleaseEngineering/How_To/Modify_scripts_on_stage

Comment 12

[Ed Morley [:emorley]]

(In reply to Nick Thomas [:nthomas] (away Dec 21 - Jan 3) from comment #11)
> If this is wanted before I get back early January then please contact #build
> on irc. 

There is no rush on this at all - enjoy your holiday :-)

Comment 13

[[e-mail] (:bhearsum)]

This doesn't look like a buildduty item. Also, is there anything left to do here?

Comment 14

[Phil Ringnalda (:philor)]

The thing left to do is comment 11, revert back to ftp. from ftp-scl3.

Comment 15

[Ed Morley [:emorley]]

Buildduty since this impacts builds, eg:
https://tbpl.mozilla.org/php/getParsedLog.php?id=18641972&tree=Firefox

The temporary fix here is actually now less reliable.

The attached patch here needs reverting (see bug 811063 comment 81).

Comment 16

[Ed Morley [:emorley]]

(From SVN - otherwise I'd do it :-))

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Ed Morley [:edmorley UTC+0] from comment #15)
> The temporary fix here is actually now less reliable.

That's a 500 error from the ftp cluster, doesn't matter if you get there via DNS path that stays inside Mozilla or not. Most likely it was a result of an Apache config change on the cluster, which doesn't get handled very gracefully because of the high load. Bug 826495 tracks improving that.

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 686313 [details] [diff] [review]
[svn] Swap to ftp-scl3.m.c

If we're going to do this we'd better do it while Berlin is asleep, and before the 15th gets too much closer.

Committed revision 55928.

Comment 19

[Nick Thomas [:nthomas] (UTC+12)]

Puppet deployed the change at Wed Jan 9 18:24:53 PST 2013. Uploads happening after that will use ftp.m.o.

Comment 20

[Ed Morley [:emorley]]

Thank you, looks good :-)