Closed Bug 1403922 Opened 8 years ago Closed 8 years ago

Investigate beetmover slowdown

Categories

(Release Engineering :: General, defect, P1)

Product:
Release Engineering
Component:
General
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: catlee, Unassigned)

References

Details

At 13:02 UTC today, we had some strange delays doing chain of trust validation on beetmover-3. From papertrail: https://papertrailapp.com/systems/730136041/events?focus=850045034156953613&selected=850045034156953613 Sep 28 06:02:31 beetmoverworker-3.srv.releng.use1.mozilla.com python: beetmover_scriptworker 2017-09-28T13:02:31 INFO - Done Sep 28 06:02:31 beetmoverworker-3.srv.releng.use1.mozilla.com python: beetmover_scriptworker 2017-09-28T13:02:31 DEBUG - /builds/scriptworker/work/cot/dW3BUPF0TRqOEBlRzd27nw/public/build/as/target.tar.bz2 matches the expected sha256 07cd775eb6f2cb8a52f23800773d428286605dfd807cbd801ea5d501697312f6 Sep 28 06:03:26 beetmoverworker-3.srv.releng.use1.mozilla.com sudo: nagios : TTY=unknown ; PWD=/ ; USER=cltbld ; COMMAND=/usr/lib64/nagios/plugins/check_file_age -w 2700 -c 3600 -f /builds/scriptworker/logs/worker.log Sep 28 06:03:34 beetmoverworker-3.srv.releng.use1.mozilla.com collectd: ethstat plugin: No stats available for eth0 Sep 28 06:07:17 beetmoverworker-3.srv.releng.use1.mozilla.com python: beetmover_scriptworker 2017-09-28T13:07:17 DEBUG - Reclaiming task... From the chain of trust log https://public-artifacts.taskcluster.net/OwYsx-eWSqKGdpqPJc9wFA/0/public/logs/chain_of_trust.log : 2017-09-28T13:02:30 DEBUG - /builds/scriptworker/work/cot/cpqgOMoNQCaYOomXoa6mVw/public/build/as/target.complete.mar matches the expected sha256 24882599e25259c3135c67b0759b415382a74135a67faa7eb03da3093a6bb835 2017-09-28T13:02:31 DEBUG - /builds/scriptworker/work/cot/dW3BUPF0TRqOEBlRzd27nw/public/build/as/target.tar.bz2 matches the expected sha256 07cd775eb6f2cb8a52f23800773d428286605dfd807cbd801ea5d501697312f6 2017-09-28T13:07:39 DEBUG - /builds/scriptworker/work/cot/fDH1vQ1IRGG39DD5YgRzRQ/public/build/as/target.langpack.xpi matches the expected sha256 362688c01ccb49c374ccdca40e57308d3413589a48f3c817cec8958328bc146b 2017-09-28T13:07:39 INFO - Verifying beetmover OwYsx-eWSqKGdpqPJc9wFA as a beetmover task... Looking at the EC2 instance's cloudwatch metrics shows that CPU and network usage both drop to close to 0 around this time. However, there are still 100s of CPU credits.
See Also: → 1402746
Priority: -- → P1
https://github.com/mozilla-releng/scriptworker/pull/159 should help with overall throughput, but doesn't explain this particular slowdown. Simon, Mihai, is there anything else we wanted to do here?
Flags: needinfo?(sfraser)
Component: General → General Automation
QA Contact: catlee
Nothing comes to head offhand. I should look into recent beetmover jobs to see what's going on. FTR, we've also upgraded the instance types in the meantime.
Nothing from me at the moment, either.
Flags: needinfo?(sfraser)
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → WORKSFORME
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.