927512 - Reduce the amount of talos testing on long-term support branches

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

Joel and I were discussing this in #ateam today. We currently run the full talos suite of tests on the b2g18* and ESR branches, even though the value of doing so is likely very limited (i.e. are we really going to back out a critical security update from b2g18 if it regresses JS perf by 2% on Windows?).

Additionally, Windows talos on b2g18/esr17 is littered with frequent test failures like the one below:
https://tbpl.mozilla.org/php/getParsedLog.php?id=29199958&tree=Mozilla-B2g18

Traceback (most recent call last):
  File "C:\talos-slave\talos-data\talos\bcontroller.py", line 226, in ?
    sys.exit(main())
  File "C:\talos-slave\talos-data\talos\bcontroller.py", line 223, in main
    bcontroller.run()
  File "C:\talos-slave\talos-data\talos\bcontroller.py", line 168, in run
    raise talosError(str(e))
utils.talosError: "[Errno 13] Permission denied: 'browser_output.txt'"
Failed dromaeo_css: 
		Stopped Wed, 16 Oct 2013 10:49:21
Traceback (most recent call last):
  File "run_tests.py", line 249, in run_tests
    talos_results.add(mytest.runTest(browser_config, test))
  File "C:\talos-slave\talos-data\talos\ttest.py", line 370, in runTest
    raise talosError("timeout exceeded")
talosError: 'timeout exceeded'
Traceback (most recent call last):
  File "run_tests.py", line 295, in ?
    main()
  File "run_tests.py", line 292, in main
    run_tests(parser)
  File "run_tests.py", line 256, in run_tests
    raise e
utils.talosError: 'timeout exceeded'

So I would propose the following as a bare-minimum (nothing of value lost):
* esr17: Don't do anything. It'll be dead soon enough anyway.
* b2g18*: Disable talos on Windows and OSX. Leave them running on Linux since it's the most B2G-like platform.
* esr24: Disable talos on OSX (at least). Arguably, Windows is the primary use case for the ESR releases, but since it's also known to be used in various Linux LTS distributions, might as well leave it there too.

I would also urge considering disabling talos outright on these branches as I think the reality is that running it offers little to no value. I'll point out that none of these even report to dev.tree-management.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

I would second this proposal.  While these are not high volume branches, we should be smart about what we are testing.

 * b2g18 is for b2g builds only- this is gecko on a 32 bit linux type platform- windows and osx do not really matter.  On a related note, would disabling unittests be wise?

 * esr24 - this is nice to have full support, I would like to know what users run esr24 (we could do esr17 for a larger sample size maybe).  If we have 99%+ of our users on windows, it makes sense to turn off performance tests for osx and linux.

Please weigh in here, we should post on the appropriate newsgroups and bring this up in a platform meeting prior to making a change of this magnitude.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

The proportion of windows users is about 92% for all of release, esr-24, and esr-17 (based on blocklist pings, 15 Oct). There is less than one percentage point difference between them.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

This would indicate we shouldn't touch ESR releases, although we should consider updating talos when a major fix solves issues that cause frequent failures.  As RyanVM mentioned, lets not change or update anything on esr17, but lets focus on esr24 as an opportunity to update talos if needed.  A lot of little fixes have changed to make Talos more robust.  Also it appears that the test slaves have been receiving updates which means even if the build and toolchain doesn't change, we could still get failures and have to update the code we control.

Comment 4

[Andrew Overholt [:overholt]]

(In reply to Ryan VanderMeulen [:RyanVM UTC-4][PTO 10/19 - 10/27] from comment #0)
> * b2g18*: Disable talos on Windows and OSX.

+1 (I'm not going to comment on the esr discussion since I'm not as familiar with those branches)

Comment 5

[Chris AtLee [:catlee]]

Attachment: disable windows and osx talos on b2g18* branches

Comment 6

[Chris AtLee [:catlee]]

We should also consider shutting off non-linux builds and unittests.

Comment 7

[Rail Aliiev [:rail]]

Comment on attachment 824187 [details] [diff] [review]
disable windows and osx talos on b2g18* branches

DIAF!

Comment 8

[Ryan VanderMeulen [:RyanVM]]

(In reply to Chris AtLee [:catlee] from comment #6)
> We should also consider shutting off non-linux builds and unittests.

At a bare minimum, I would think running builds/tests on only one flavor of OSX and Windows would be an easy win.

Comment 9

[Ryan VanderMeulen [:RyanVM]]

Well, OSX anyway. Windows tests all run on rev3s, so killing them probably won't be of much help elsewhere.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

in production

Comment 11

[Ryan VanderMeulen [:RyanVM]]

We should probably disable talos for OSX/Windows on b2g26 as well.

Comment 12

[Ryan VanderMeulen [:RyanVM]]

Attachment: disable windows and osx talos on b2g26_v1_2 branch

Comment 13

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 827637 [details] [diff] [review]
disable windows and osx talos on b2g26_v1_2 branch

https://hg.mozilla.org/build/buildbot-configs/rev/086d4e03a4e0

Comment 14

[Aki Sasaki (not active)]

In production.

Comment 15

[Ryan VanderMeulen [:RyanVM]]

Based on the comments, I believe this bug is done. We left the ESR branches as-is based on usage statistics being the same as release and killed OSX/Windows talos on the B2G release branches.

Fundamentally, I still wonder about the utility of caring about talos regressions on ESR branches (and whether anybody is even looking at them), but we can save that debate for another day :)