re-spin up signing-linux-*, depsigning-worker* as t2.micro

RESOLVED FIXED

Status

RESOLVED FIXED
a year ago
a year ago

People

(Reporter: aki, Assigned: aki)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Assignee)

Description

a year ago
With the old pefile, checking for win sigs took >1.5gb ram, and we were timing out all over on t2.micro instances. I bumped the signing-linux-* and depsigning-worker* instances to t2.medium, which fixed the immediate symptoms.

Catlee landed https://github.com/mozilla-releng/signtool/pull/4 which significantly speeds up and reduces the memory footprint for win sig verification, so we should be able to spin them back down to t2.micro.
(Assignee)

Comment 1

a year ago
I already did this, then reverted the depsigning pool to t2.medium in bug 1384667.
Looks like we don't need this -- the issue was a misconfigured signing6, rather than a worker-side issue.

I reverted depsigning-worker{2,4,6,8,10,12,14,16} (even numbered) to t2.micro. The odd-numbered depsigning-workers are t2.medium.
If we're good overnight, let's change the odd-numbered depsigning-workers to t2.micro and resolve :)
Assignee: nobody → aki
(Assignee)

Updated

a year ago
Depends on: 1293783
I see the CPU credits/hour drops by a factor of 4 to 6 when dropping to t2.micro. Do we have metrics for how long jobs take which might tell us if the difference between baseline and burst CPU is significant ?
(Assignee)

Comment 3

a year ago
We seem to have a decent amount of cpu creds on t2.micro. I don't have a lot of metrics, just eyeballing when we're running jobs. We seem to be able to sign somewhere between 2-3min on average and that can go up to 5-8min on t2.micro. Both times we noticed that t2.micro instances were going over the 30min mark were when something else was wrong: 1st time pefile used 1.5gb ram to see if xul.dll was signed, and 2nd time signing6 was timing out and retrying on all windows signing.

Right now all the even-numbered (usw2) instances are t2.micro and odd-numbered (use1) are t2.medium, so we might be able to compare.
(Assignee)

Comment 4

a year ago
All t2.micro.
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.