1381639 - re-spin up signing-linux-*, depsigning-worker* as t2.micro

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, enhancement)

Description

[Aki Sasaki (not active)]

With the old pefile, checking for win sigs took >1.5gb ram, and we were timing out all over on t2.micro instances. I bumped the signing-linux-* and depsigning-worker* instances to t2.medium, which fixed the immediate symptoms.

Catlee landed https://github.com/mozilla-releng/signtool/pull/4 which significantly speeds up and reduces the memory footprint for win sig verification, so we should be able to spin them back down to t2.micro.

Comment 1

[Aki Sasaki (not active)]

I already did this, then reverted the depsigning pool to t2.medium in bug 1384667.
Looks like we don't need this -- the issue was a misconfigured signing6, rather than a worker-side issue.

I reverted depsigning-worker{2,4,6,8,10,12,14,16} (even numbered) to t2.micro. The odd-numbered depsigning-workers are t2.medium.
If we're good overnight, let's change the odd-numbered depsigning-workers to t2.micro and resolve :)

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

I see the CPU credits/hour drops by a factor of 4 to 6 when dropping to t2.micro. Do we have metrics for how long jobs take which might tell us if the difference between baseline and burst CPU is significant ?

Comment 3

[Aki Sasaki (not active)]

We seem to have a decent amount of cpu creds on t2.micro. I don't have a lot of metrics, just eyeballing when we're running jobs. We seem to be able to sign somewhere between 2-3min on average and that can go up to 5-8min on t2.micro. Both times we noticed that t2.micro instances were going over the 30min mark were when something else was wrong: 1st time pefile used 1.5gb ram to see if xul.dll was signed, and 2nd time signing6 was timing out and retrying on all windows signing.

Right now all the even-numbered (usw2) instances are t2.micro and odd-numbered (use1) are t2.medium, so we might be able to compare.

Comment 4

[Aki Sasaki (not active)]

All t2.micro.