zeus: upgrade from 8.0r0 to 8.1r1

RESOLVED FIXED

Status

RESOLVED FIXED
7 years ago
6 years ago

People

(Reporter: Atoll, Assigned: ckolos)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [qa+][needs loadtest])

(Reporter)

Description

7 years ago
Upgrade from Zeus 8.0 to anything newer in the 8.x branch. We're experiencing memory corruption in 8.0 (PHX1) and need to resolve this.

Stage first, then Prod when possible.

Concurrently, we're going to do the INIT_CWND kernel updates. Once those updates have been deployed to staging, recompile the multihost kernel module and ensure that VIPs still work correctly.
(Reporter)

Comment 1

7 years ago
adm1.scl2.stage.svc:~rsoderberg/zeustm_81r1_linux-x86_64.tgz ready for deployment
Summary: zeus: upgrade from 8.0 to 8.x → zeus: upgrade from 8.0r0 to 8.1r1
(Reporter)

Comment 2

7 years ago
and zeus_modules_installer-2.4.tgz for the kernel module update
Assignee: nobody → rsoderberg
(Reporter)

Updated

7 years ago
Depends on: 778973
No longer depends on: 732056
Whiteboard: [qa+]
(Reporter)

Comment 3

6 years ago
Stage upgraded from 8.0r0 to 8.1r1. Holding for loadtest before proceeding with production. Total upgrade process is:

# tar -xzvf zeustm_81r1_linux-x86_64.tgz && cd ZeusTM_81r1_Linux-x86_64 && ./zinstall
(Reporter)

Updated

6 years ago
Whiteboard: [qa+] → [qa+][needs loadtest]
:atoll any product in particular? Or just some kind of load test in Stage (BID, Sync, etc) ?.....
Status: NEW → ASSIGNED
QA Contact: jbonacci
I started a 250M ADU browserid load_gen now. That's not a super heavy load test (it will do about 60 https qps on the webheads, 25 https qps on verifier (also front LB), and less on swebs, keysigner (back)). I can bring up more if you would prefer.
s/250M/250K/. ;-)
In general not having issues yet. Let me know if you want more traffic. 

I had one ETIMEDOUT on a socket at 2012-08-22 16:53:53 PDT. Do you see anything in the zeus logs at that time. I don't see anything specific on the server side node logs. (These chirps happen occasionally, but I've not had success in pinning it down to a particular cause).
(Reporter)

Updated

6 years ago
Blocks: 749315
At this point, the test is still running, and I have only had that one chirp so far.
:atoll sounds like we can call this good.
Just saw this in production on my firefox sync account.
(Reporter)

Comment 11

6 years ago
Bug 807765 is tracking the Zeus 8.1r1 upgrade for an unrelated reason, and once that's done we'll check back here to make sure that's resolved all of the things.
Assignee: rsoderberg → nobody
Depends on: 807765
(Reporter)

Updated

6 years ago
Blocks: 807765
No longer depends on: 807765
Assignee: nobody → ckolos
(Assignee)

Comment 12

6 years ago
all zlb's have been patched to 8.1r1
Status: ASSIGNED → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.