1672295 - js-bench tests are timing out during hg clone

Status: RESOLVED FIXED

(Testing :: Performance, defect, P1)

Description

[Andrew Erickson [:aerickson]]

These test jobs are timing out while doing the mercurial robustcheckout on the new 1804 workers I'm trying to start up. They're taking more than the 30 minutes allowed in the job configuration.

I've chatted with Connor (in Slack #vcs) and Chris B (in Slack #netops) to debug the slow clone. The robustcheckout is fetching from Cloudfront/Amazon. DPI is not enabled on the router in MDC2. Connor mentioned Bug 1670930, but still didn't think the operation should be this slow.

Cloning on my home internet connection (100Mb/sec connection) took 40 minutes to clone with the same command as the test jobs. hg robustcheckout --purge --sharebase /tmp/testdir --upstream https://hg.mozilla.org/mozilla-unified --revision e9df3ce8632c777f20058835e92e9c1b99af23ed https://hg.mozilla.org/try /tmp/testdir

example treeherder job description: Job: Linux x64 shippable opt JavaScript shell benchmarks with Google V8 source-test-jsshell-bench-sixspeed-v8 6speed

mach try fuzzy syntax that will run the jobs:
./mach try fuzzy --full --query "'source-test-jsshell-bench" --no-push

Comment 1

[Andrew Erickson [:aerickson]]

Attachment: live_backing.log

Failing job log output.

Comment 2

[Andrew Erickson [:aerickson]]

Discussed with jmaher and gmierz over email. I'll push a phab shortly to increase the timeout and we'll file a bug to track down the cause of the slow clone.

Comment 3

[Andrew Erickson [:aerickson]]

Attachment: Bug 1672295: increase max-run-time for js-bench jobs

Increase max-run-time to one hour.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b253a44b5b43
increase max-run-time for js-bench jobs r=jmaher

Comment 5

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/b253a44b5b43