1676487 - Reduce max-run-time for Marionette jobs in CI

Status: RESOLVED FIXED

(Remote Protocol :: Marionette, task, P1)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 07/21 - 08/08)]

As discovered https://phabricator.services.mozilla.com/D96466#inline-544594 we use a max-run-time of 5400s at the moment. That is quite a lot for the Marionette jobs.

We should reduce it for all Mn jobs to a lower value. I will check what our usual runtime is for the job across platforms, and if we simply can reduce the timeout, or maybe have to divide the jobs into chunks.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 07/21 - 08/08)]

Joel, does one of you have a tool that can scrape jobs of specific types on Treeherder/Taskcluster and fetch their duration? Doing that manually is a bit bothersome, and I could imagine that also other job types could benefit from that.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

there is no simple tool that I know of, ./mach test-info .. can provide some insight into specific tests, but not really test jobs. You could query the treeherder database on redash, or use activedata as well.

I did a quick redash query using the treeherder datasource:

select
  jt.symbol,
  jt.name,
  j.end_time-j.start_time
from
  job j,
  job_type jt
where 
  j.job_type_id=jt.id
  and jt.name like '%marionette%'
  and j.result='success'
limit 100

it should get you a headstart if you want to use redash.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 07/21 - 08/08)]

As discussed on Matrix the above query doesn't work in different ways. So here an updated one:

select
  jt.name,
  j.start_time,
  j.end_time,
  timestampdiff(second, j.start_time, j.end_time) as seconds,
  jl.url
from
  job j,
  job_type jt,
  job_log jl
where
  j.job_type_id=jt.id
  and jl.job_id=j.id
  and jt.name like '%opt-marionette-e10s%'
  and jt.name not like '%ccov%'
  and jt.name not like '%devedition%'
  and j.result='success'
  and timestampdiff(second, j.start_time, j.end_time) > 2000
limit 100

Basically here what I got suggested...

• No job should run longer than 3600s
• for jobs of opt builds 1800s is fine
• for jobs of debug/asan/ccov builds 2700s - 3600s are fine

Jobs that take longer should be split into multiple chunks.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 07/21 - 08/08)]

Lets test the proposed timeouts from the last comment if those work and if not if we could chunk the Mn jobs.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 07/21 - 08/08)]

I've pushed a try build. Lets see how it works:
https://treeherder.mozilla.org/jobs?repo=try&revision=4dfdeb6dcba9365935790e8eeaf5f3c6253eb055

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 07/21 - 08/08)]

Attachment: Bug 1676487 - [marionette] Reduce max-run-time for jobs in Taskcluster.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b034b4e4cb99
[marionette] Reduce max-run-time for jobs in Taskcluster. r=jmaher

Comment 8

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/b034b4e4cb99