Upgrade from Zeus 8.0 to anything newer in the 8.x branch. We're experiencing memory corruption in 8.0 (PHX1) and need to resolve this. Stage first, then Prod when possible. Concurrently, we're going to do the INIT_CWND kernel updates. Once those updates have been deployed to staging, recompile the multihost kernel module and ensure that VIPs still work correctly.
adm1.scl2.stage.svc:~rsoderberg/zeustm_81r1_linux-x86_64.tgz ready for deployment
Summary: zeus: upgrade from 8.0 to 8.x → zeus: upgrade from 8.0r0 to 8.1r1
and zeus_modules_installer-2.4.tgz for the kernel module update
Assignee: nobody → rsoderberg
Stage upgraded from 8.0r0 to 8.1r1. Holding for loadtest before proceeding with production. Total upgrade process is: # tar -xzvf zeustm_81r1_linux-x86_64.tgz && cd ZeusTM_81r1_Linux-x86_64 && ./zinstall
Whiteboard: [qa+] → [qa+][needs loadtest]
:atoll any product in particular? Or just some kind of load test in Stage (BID, Sync, etc) ?.....
Status: NEW → ASSIGNED
QA Contact: jbonacci
I started a 250M ADU browserid load_gen now. That's not a super heavy load test (it will do about 60 https qps on the webheads, 25 https qps on verifier (also front LB), and less on swebs, keysigner (back)). I can bring up more if you would prefer.
In general not having issues yet. Let me know if you want more traffic. I had one ETIMEDOUT on a socket at 2012-08-22 16:53:53 PDT. Do you see anything in the zeus logs at that time. I don't see anything specific on the server side node logs. (These chirps happen occasionally, but I've not had success in pinning it down to a particular cause).
At this point, the test is still running, and I have only had that one chirp so far.
:atoll sounds like we can call this good.
Just saw this in production on my firefox sync account.
Bug 807765 is tracking the Zeus 8.1r1 upgrade for an unrelated reason, and once that's done we'll check back here to make sure that's resolved all of the things.
Assignee: rsoderberg → nobody
Depends on: 807765
all zlb's have been patched to 8.1r1
Status: ASSIGNED → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.