Closed Bug 774354 Opened 14 years ago Closed 13 years ago

[tracker] move from single relengweb1 to a real-live webapp cluster

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: dustin, Assigned: dustin)

References

(
URL
)

Details

(Whiteboard: [2013Q2])

Attachments

(1 file)

relengweb_verify.sh
10.66 KB, text/plain
Details
Cluster IPs: bug 772267 VMs: bug 774350, bug 772269 This change should be transparent to users of the service, but gets us better resiliency, better support (by taking the same form as other services), and room to grow.
Assignee: server-ops-releng → dustin
Depends on: 604688
I set up a new VIP (releng-zlb.vips.scl3.mozilla.com) and zeus configuration pointing only to web2. I didn't set up releng-https yet, since I'm not sure how we'll handle the SSL during the transition.
https://infra.etherpad.mozilla.org/86 tracks the sites that I'll need to move to this new cluster, along with my notes and todo's.
URL: https://infra.etherpad.mozilla.org/86
When bug 780659 (and a follow-on flow request) is closed, this will be ready for testing. I'll post an /etc/hosts entry here when that's the case.
Depends on: 781480
Once bug 781480 is clear, I'll hand this back to releng to verify all of these services work on the new cluster (along with instructions and an /etc/hosts snippet).
No longer depends on: 604688
Blocks: 768123
OK! I'd like to transition to this new service as soon as possible, but I want to make sure everything is picture-perfect first. I'm quite confident, given what I learned moving this to scl3, but validation is good. If someone from releng could test this out, I'd very much appreciate it! After that, I'll begin moving one service at a time to the new cluster, with no downtime required. I'll coordinate with buildduty. /etc/hosts: 10.22.74.160 runtime-binaries.pvt.build.mozilla.org tooltool.pvt.build.mozilla.org clobberer.pvt.build.mozilla.org clobberer-stage.pvt.build.mozilla.org clobberer-preproduction.pvt.build.mozilla.org talos-bundles.pvt.build.mozilla.org 63.245.215.57 trychooser.pub.build.mozilla.org clobberer.pub.build.mozilla.org clobberer-stage.pub.build.mozilla.org builddata.pub.build.mozilla.org mobile-dashboard.pub.build.mozilla.org update-tests.pub.build.mozilla.org secure.pub.build.mozilla.org tooltool.pub.build.mozilla.org Test URLs: --public-- http://update-tests.pub.build.mozilla.org/pub/mozilla.org/firefox/nightly/2.0.0.16-candidates/build1/linux_info.txt http://builddata.pub.build.mozilla.org/buildjson/ http://builddata.pub.build.mozilla.org/reports/ http://tooltool.pub.build.mozilla.org/try/TEMPORARY.txt http://clobberer.pub.build.mozilla.org http://clobberer-stage.pub.build.mozilla.org http://trychooser.pub.build.mozilla.org/ https://secure.pub.build.mozilla.org/builddata/reports/ https://secure.pub.build.mozilla.org/builddata/buildjson/ (no LDAP required) https://secure.pub.build.mozilla.org/buildapi/ https://secure.pub.build.mozilla.org/clobberer/ https://secure.pub.build.mozilla.org/clobberer-stage/ --private-- (please try these from MPT-VPN too, to check they are build-net-only) (* indicates VPN is specifically excluded; test from a slave, or cruncher) (# indicates a binary file - be careful if using curl!) http://talos-bundles.pvt.build.mozilla.org/findlinks/ http://runtime-binaries.pvt.build.mozilla.org/README.txt http://tooltool.pvt.build.mozilla.org/build/sha512/0bf5cceced8add6142c1c7522890c39554b94848e3f5eb9b92de2a8c6f6af3c0e8ab69ddf7bae39eaab2bcb637233ff02de07b47a1850767d2810a46fb31bf65 # http://clobberer.pvt.build.mozilla.org * http://clobberer-stage.pvt.build.mozilla.org * http://clobberer-preproduction.pvt.build.mozilla.org * TIA!
Assignee: dustin → nobody
Component: Server Operations: RelEng → Release Engineering: Developer Tools
QA Contact: arich → hwine
Depends on: 781928
bonus: this cluster now has two webheads in it, so we get some redundancy and ability to shift load around during maintenance.
Blocks: 787391
There was some confusion here today. The next steps for this bug are for releng to sign off on putting it into production. The degree of investigation is up to you. I'd recommend at least hitting the URLs above and verifying that the results make sense, but any additional functionality testing will be great, too. I'd like to get the whole thing verified before moving anything to it.
Blocks: 804334
Blocks: 841345
Blocks: 855733
Blocks: 796916
Whiteboard: [2013Q2]
Blocks: 773057
Depends on: 864449
Depends on: 864450
Depends on: 864453
Depends on: 864454
Depends on: 864455
Depends on: 864458
Depends on: 864460
Depends on: 864466
Depends on: 864469
Depends on: 864482
Depends on: 864486
Attached file relengweb_verify.sh —
I wrote this up to test access works as expected for both clusters in a variety of contexts: external - outside of Mozilla vpn - connected via vpn internal - inside Mozilla, but not on the releng network relengnet - on the releng network I couldn't find any hosts that met the definition of "internal" that had flows into the releng network. Which is as it should be, but prevented testing that context. Otherwise, this is all "PASS" and "SKIP" from everywhere I ran it. Note that this does not use DNS, so rather than being a tool to run before and after any change, this proves to my satisfaction *now* that the DNS change will have no user-visible effect. As an example: --- vpn ---- == testing old cluster == PASS cleartext builds-pending.js PASS cleartext last-job-per-slave PASS secure builds-pending.js PASS secure last-job-per-slave PASS clobberer.pub.bmo redir PASS clobberer-stage.pub.bmo redir SKIP clobberer-preproduction.pub.bmo redir on old PASS clobberer UI requires auth PASS clobberer-stage UI requires auth PASS authenticated clobberer UI works PASS authenticated clobberer-stage UI works PASS unauthenticated clobberer API works PASS unauthenticated clobberer-stage API works PASS unauthenticated clobberer-preproduction API works PASS mobile-dashboard PASS runtime-binaries PASS talos-bundles SKIP slavealloc not implemented on old SKIP slavealloc not implemented on old SKIP slavealloc not implemented on old PASS tooltool.pub.bmo PASS tooltool.pvt.bmo PASS trychooser.pub.bmo PASS update-tests.pub.bmo PASS secure.pub.bmo/aws-support has a CRL in it PASS secure.pub.bmo/buildapi requires auth PASS secure.pub.bmo/buildapi seems to proxy properly == testing new cluster == PASS cleartext builds-pending.js PASS cleartext last-job-per-slave PASS secure builds-pending.js PASS secure last-job-per-slave PASS clobberer.pub.bmo redir PASS clobberer-stage.pub.bmo redir PASS clobberer-preproduction.pub.bmo redir PASS clobberer UI requires auth PASS clobberer-stage UI requires auth PASS authenticated clobberer UI works PASS authenticated clobberer-stage UI works PASS unauthenticated clobberer API works PASS unauthenticated clobberer-stage API works PASS unauthenticated clobberer-preproduction API works PASS mobile-dashboard PASS runtime-binaries PASS talos-bundles PASS secure.pub.bmo/slavealloc should require auth PASS secure.pub.bmo/slavealloc should serve the main page with auth PASS slavealloc API PASS tooltool.pub.bmo PASS tooltool.pvt.bmo PASS trychooser.pub.bmo PASS update-tests.pub.bmo PASS secure.pub.bmo/aws-support has a CRL in it PASS secure.pub.bmo/buildapi requires auth PASS secure.pub.bmo/buildapi seems to proxy properly
Assignee: nobody → dustin
Attachment #759468 - Attachment mime type: application/x-sh → text/plain
DNS changed at 10am PDT today with no issues.
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
Component: Tools → General
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: