778944 - zeus: upgrade from 8.0r0 to 8.1r1

Status: RESOLVED FIXED

(Cloud Services :: Operations: Miscellaneous, task)

Description

[:Atoll]

Upgrade from Zeus 8.0 to anything newer in the 8.x branch. We're experiencing memory corruption in 8.0 (PHX1) and need to resolve this.

Stage first, then Prod when possible.

Concurrently, we're going to do the INIT_CWND kernel updates. Once those updates have been deployed to staging, recompile the multihost kernel module and ensure that VIPs still work correctly.

Comment 1

[:Atoll]

adm1.scl2.stage.svc:~rsoderberg/zeustm_81r1_linux-x86_64.tgz ready for deployment

Comment 2

[:Atoll]

and zeus_modules_installer-2.4.tgz for the kernel module update

Comment 3

[:Atoll]

Stage upgraded from 8.0r0 to 8.1r1. Holding for loadtest before proceeding with production. Total upgrade process is:

# tar -xzvf zeustm_81r1_linux-x86_64.tgz && cd ZeusTM_81r1_Linux-x86_64 && ./zinstall

Comment 4

[James Bonacci [:jbonacci]]

:atoll any product in particular? Or just some kind of load test in Stage (BID, Sync, etc) ?.....

Comment 5

[John Morrison [:jrgm]]

I started a 250M ADU browserid load_gen now. That's not a super heavy load test (it will do about 60 https qps on the webheads, 25 https qps on verifier (also front LB), and less on swebs, keysigner (back)). I can bring up more if you would prefer.

Comment 6

[John Morrison [:jrgm]]

s/250M/250K/. ;-)

Comment 7

[John Morrison [:jrgm]]

In general not having issues yet. Let me know if you want more traffic. 

I had one ETIMEDOUT on a socket at 2012-08-22 16:53:53 PDT. Do you see anything in the zeus logs at that time. I don't see anything specific on the server side node logs. (These chirps happen occasionally, but I've not had success in pinning it down to a particular cause).

Comment 8

[John Morrison [:jrgm]]

At this point, the test is still running, and I have only had that one chirp so far.

Comment 9

[James Bonacci [:jbonacci]]

:atoll sounds like we can call this good.

Comment 10

[Gene Wood [:gene]]

Just saw this in production on my firefox sync account.

Comment 11

[:Atoll]

Bug 807765 is tracking the Zeus 8.1r1 upgrade for an unrelated reason, and once that's done we'll check back here to make sure that's resolved all of the things.

Comment 12

[Chris Kolosiwsky [:ckolos] (ckolos has left the building)]

all zlb's have been patched to 8.1r1