Open Bug 1573872 Opened 5 months ago Updated 10 days ago

[meta] reduce test jobs to be run more appropriately


(Testing :: General, task, P3)

Version 3


(Not tracked)


(Reporter: jmaher, Unassigned)


(Depends on 8 open bugs, Blocks 1 open bug)


(Keywords: meta)

  1. consider web-platform-tests that are not passing to run once/day on m-c
  2. consider mochitest that are not passing to run once/day on m-c
  3. consider changes to webrender to run on cheaper instances/non-gpu
  4. consider webrender scheduling to be gfx/* only
  5. consider devtools tests to only run on devtools/*
  6. help interop team reduce duplicate mochitest/reftest jobs with wpt
  7. reduce load on perf pageload tests by looking for duplicate patterns and only running duplicates once/day

turn off AWSY on ESR branches.

Depends on: 1564487
Priority: -- → P3
See Also: → test-efficiencies

another vector to consider is tests/jobs that never fail- if we could have data on test cases that have not failed in the last year (maybe too long), we could consider running those tests as tier-2. Possibly a solution is to say if a test fails (intermittent/perma, but not random) that we run it and all tests in the same directory as tier-1. That changes a lot of scheduling dynamics...worth discussing.

Depends on: 1598744
Depends on: 1581194
Depends on: 1598746

4 things that are identified as worth starting in December 2019:

  • [:ahal] Bug 1581194 - SETA every 10/120 instead of 5/60
  • [TODO] Bug 1598744 - Stop running tier2 and tier3 jobs on autoland and try by default
  • [:bc] Bug 1572820 - Reduce WPT usage by running less relevant tests to Firefox once/day on MC only, not autoland/try by default
  • [TODO] Bug 1598746 - do not run default set of tasks for commits that are not firefox code related
Depends on: 1599197

Last month on autoland we spent $115K
$78,449 = tier1
$25,012 = tier2
$12,163 = tier3

data from Oct 21 -> Nov 21

a 5th item:

  • [TODO] Bug 1599197 - demote linux32 builds to tier-2 (not run on autoland and by default on try)

cost savings:
[:ahal] Bug 1581194 - seta - $175K (20% realized, max is 30%, with backfilling/retriggers, I see 20% as realistic)
[:bc] Bug 1572820 - wpt reduction - $50K (10% savings accounting for SETA and some tests that are providing value despite results)
[:gbrown] Bug 1598746 - DONTBUILD for select commits - $125K (alone $175K with a 20% drop, including above , reduced $50K)
Bug 1598744 - tier2/3 - $100K (see bug for details, ignoring above wins)
Bug 1599197 - demote linux32 builds - $37K

Total savings $487K/year. I still need to find owners for the last 2 bugs

I'll put my name up for either one of the last two. I feel bug 1599197 is too small to count as one task (unless there is some complications I'm not seeing, famous last words).

My preference order would be:
Bug 1599197
Bug 1598744

ok, Bug 1599197 is assigned to :egao, we will circle back in a couple weeks to see who can take Bug 1598744

Depends on: 1490969
Depends on: 1604547, 1604541
Depends on: 1607447
Depends on: 1607619
Depends on: 1607803

After looking at autoland a lot over the last week or so, I have a few more items that are curious:

  • gcp builds: full set of these are triggered for every autoland commit, and they are tier 3.
  • puppeteer: tier 3 and run all the time, and they always fail.
You need to log in before you can comment on or make changes to this bug.