1577427 - Intermittent mbu python [taskcluster:error] Task aborted - max run time exceeded

Status: RESOLVED INCOMPLETE

(Developer Infrastructure :: Lint and Formatting, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: apavel [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=263990422&repo=mozilla-inbound
Full log: https://queue.taskcluster.net/v1/task/H8IbohL6RpWEdzPCV9WiVg/runs/0/artifacts/public/logs/live_backing.log

---

vcs 2019-08-29T05:16:54.089Z] updating [=========> ] 63400/283207 1m28s
[taskcluster:error] Aborting task...
[taskcluster:warn 2019-08-29T05:16:54.368Z] no such process
[taskcluster 2019-08-29T05:16:54.393Z] === Task Finished ===
[taskcluster 2019-08-29T05:16:54.393Z] Task Duration: 1h0m0.169719513s
[taskcluster 2019-08-29T05:16:54.393Z] [mounts] Preserving cache: Moving "/Users/cltbld/tasks/task_1567051777/checkouts" to "/Users/cltbld/caches/FrvyMJLITeCDngu3LMjwxA"
[taskcluster:error] Task aborted - max run time exceeded

Comment 1

[Mike Hommey [:glandium]]

[vcs 2019-08-29T04:16:54.320Z] fetching hgmointernal config from http://taskcluster/secrets/v1/secret/project/taskcluster/gecko/hgmointernal
[vcs 2019-08-29T04:16:54.335Z] Unable to retrieve hgmointernal config using the secret service; falling back to public hg.mozilla.org service
[vcs 2019-08-29T04:16:54.339Z] executing ['hg', 'robustcheckout', '--sharebase', '/Users/cltbld/tasks/task_1567051777/checkouts/hg-shared', '--purge', '--upstream', 'https://hg.mozilla.org/mozilla-unified', '--revision', 'd08cf6ecb122743ab3eb567c52f37df956721f92', 'https://hg.mozilla.org/integration/mozilla-inbound', '/Users/cltbld/tasks/task_1567051777/checkouts/gecko']
[vcs 2019-08-29T04:16:54.611Z] (using Mercurial 5.1)
[vcs 2019-08-29T04:16:54.611Z] ensuring https://hg.mozilla.org/integration/mozilla-inbound@d08cf6ecb122743ab3eb567c52f37df956721f92 is available at /Users/cltbld/tasks/task_1567051777/checkouts/gecko
[vcs 2019-08-29T04:16:54.841Z] (cloning from upstream repo https://hg.mozilla.org/mozilla-unified)
[vcs 2019-08-29T04:16:54.895Z] (sharing from new pooled repository 8ba995b74e18334ab3707f27e9eb8f4e37ba3d29)
[vcs 2019-08-29T04:16:55.087Z] applying clone bundle from https://hg.cdn.mozilla.net/mozilla-unified/d3fe24ea630981ac7b2fe46912a03aff2665e1b6.zstd-max.hg

It doesn't help that it's getting the bundle from the CDN, and it helps even less that it's not taking the streaming bundle, so it spent a very long time cloning...

Comment 2

[Intermittent Failures Robot]

2 failures in 5047 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-inbound: 1
• autoland: 1

Platform breakdown:

• macosx1014-64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1577427&startday=2019-08-26&endday=2019-09-01&tree=all

Comment 3

[Connor Sheehan [:sheehan]]

Our use of streaming clone bundles in CI is enforced on the server side of hgmo - we gather the AWS ip-ranges.json file every few hours and store it server side. When a request comes in to hgmo, we check if the request originated from an IP address in one of the advertised blocks, and prioritize stream clone bundles served from the same region if we get a match.

I noticed this is a gecko-t-osx-1014 worker, which runs in MDC1. This is why the stream clone bundle isn't being used.

We have code to scrape Mozilla office-space IP address blocks in a similar fashion to the AWS IP blocks. This is supposed to be used for prioritizing stream clone bundles to offices. We could expand the query to include IP blocks for the DCs where we run Firefox CI, so stream bundles are used there as well.

Comment 4

[BugBot [:suhaib / :marco]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.