Closed
Bug 1381639
Opened 7 years ago
Closed 7 years ago
re-spin up signing-linux-*, depsigning-worker* as t2.micro
Categories
(Release Engineering :: Release Automation: Other, enhancement)
Release Engineering
Release Automation: Other
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: mozilla, Assigned: mozilla)
References
Details
With the old pefile, checking for win sigs took >1.5gb ram, and we were timing out all over on t2.micro instances. I bumped the signing-linux-* and depsigning-worker* instances to t2.medium, which fixed the immediate symptoms.
Catlee landed https://github.com/mozilla-releng/signtool/pull/4 which significantly speeds up and reduces the memory footprint for win sig verification, so we should be able to spin them back down to t2.micro.
Assignee | ||
Comment 1•7 years ago
|
||
I already did this, then reverted the depsigning pool to t2.medium in bug 1384667.
Looks like we don't need this -- the issue was a misconfigured signing6, rather than a worker-side issue.
I reverted depsigning-worker{2,4,6,8,10,12,14,16} (even numbered) to t2.micro. The odd-numbered depsigning-workers are t2.medium.
If we're good overnight, let's change the odd-numbered depsigning-workers to t2.micro and resolve :)
Assignee: nobody → aki
Comment 2•7 years ago
|
||
I see the CPU credits/hour drops by a factor of 4 to 6 when dropping to t2.micro. Do we have metrics for how long jobs take which might tell us if the difference between baseline and burst CPU is significant ?
Assignee | ||
Comment 3•7 years ago
|
||
We seem to have a decent amount of cpu creds on t2.micro. I don't have a lot of metrics, just eyeballing when we're running jobs. We seem to be able to sign somewhere between 2-3min on average and that can go up to 5-8min on t2.micro. Both times we noticed that t2.micro instances were going over the 30min mark were when something else was wrong: 1st time pefile used 1.5gb ram to see if xul.dll was signed, and 2nd time signing6 was timing out and retrying on all windows signing.
Right now all the even-numbered (usw2) instances are t2.micro and odd-numbered (use1) are t2.medium, so we might be able to compare.
Assignee | ||
Comment 4•7 years ago
|
||
All t2.micro.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•