Closed Bug 1387377 Opened 2 years ago Closed 2 years ago

Ensure we don't run microbenchmark/awsy performance numbers on devedition builds

Categories

(Testing :: Talos, enhancement)

57 Branch
enhancement
Not set

Tracking

(firefox56 fixed, firefox57 fixed)

RESOLVED FIXED
mozilla57
Tracking Status
firefox56 --- fixed
firefox57 --- fixed

People

(Reporter: igoldan, Assigned: rwood)

Details

(Whiteboard: [PI:August])

Attachments

(1 file)

From bug 1385903, it looks like platform micro values from devedition builds are miswired.
I think we can probably stop running gtest on devedition, at the very least stop reporting numbers from gtest on devedition.  I am not sure from inside the gtest how to determine if we are in devedition or not, so that would be something to look into.
Summary: Ensure we don't run performance numbers on devedition builds → Ensure we don't run microbenchmark performance numbers on devedition builds
Whiteboard: [PI:August]
I see we need to resolve this for awsy as well!

Once we successfully land this on trunk, it will need to end up on mozilla-beta to be effective :)
Summary: Ensure we don't run microbenchmark performance numbers on devedition builds → Ensure we don't run microbenchmark/awsy performance numbers on devedition builds
I believe such a check could be added here, dependent upon being able to determine if we are running on devedition somehow:

https://dxr.mozilla.org/mozilla-central/rev/a9d372645a32b8d23d44244f351639af9d73b96a/testing/gtest/rungtests.py#177
I'll see if I can get anywhere with this
Assignee: nobody → rwood
Status: NEW → ASSIGNED
Hmm, unfortunately the platform info received via mozinfo when running on Linux DevEdition opt, doesn't indicate DevEdition:


[task 2017-08-24T15:53:34.930049Z] 15:53:34     INFO -  gtest INFO | * RW * inside rungtests.py main
[task 2017-08-24T15:53:34.930528Z] 15:53:34     INFO -  gtest INFO | options:
[task 2017-08-24T15:53:34.931267Z] 15:53:34     INFO -  gtest INFO | {'symbols_path': 'https://queue.taskcluster.net/v1/task/P5KQew1WQrmskOqnsY4lFw/artifacts/public/build/target.crashreporter-symbols.zip', 'xre_path': '/home/worker/workspace/build/application/firefox', 'cwd': '/home/worker/workspace/build/tests/gtest', 'utility_path': 'tests/bin'}
[task 2017-08-24T15:53:34.932226Z] 15:53:34     INFO -  gtest INFO | mozinfo is:
[task 2017-08-24T15:53:34.932467Z] 15:53:34     INFO -  gtest INFO | {u'bin_suffix': u'', u'pgo': True, u'sync': True, u'buildapp': u'browser', u'crashreporter': True, u'addon_signing': True, u'require_signing': False, u'release_or_beta': False, u'appname': u'firefox', u'stylo': False, u'mozconfig': u'/home/worker/workspace/build/src/.mozconfig', u'topsrcdir': u'/home/worker/workspace/build/src', u'telemetry': False, 'os_version': StringVersion ('16.04'), 'version': 'Ubuntu 16.04', u'datareporting': True, u'buildtype_guess': u'pgo', 'bits': 32, 'has_sandbox': True, u'artifact': False, u'toolkit': u'gtk3', u'healthreport': True, u'updater': True, u'asan': False, u'platform_guess': u'linux', u'ccov': False, u'ubsan': False, u'tests_enabled': True, 'linux_distro': 'Ubuntu', u'official': True, u'tsan': False, u'nightly_build': True, u'debug': False, u'no_legacy_extensions': False, 'os': u'linux', 'processor': u'x86'}
[task 2017-08-24T15:53:34.932520Z] 15:53:34     INFO -  gtest INFO | Running gtest
There are DevEdition platforms specified in taskcluster test-platforms, i.e:

http://searchfox.org/mozilla-central/rev/48ea452803907f2575d81021e8678634e8067fc2/taskcluster/ci/test/test-platforms.yml#59

The solution may be to create specific test-sets in test-sets.yml, just for devedition runs, and omit gtest from there.
Comment on attachment 8900807 [details]
Bug 1387377 - Don't run gtest or awsy on devedition builds;

https://reviewboard.mozilla.org/r/172252/#review177518

I am concerned that we are adding test sets and the more we have the greater the chance of making mistakes, this is fine if there is not another option, could we edit the tests.yml file and ensure gtest doesn't run on devedition (like it is for win10 right now):
http://searchfox.org/mozilla-central/source/taskcluster/ci/test/tests.yml#226
Attachment #8900807 - Flags: review?(jmaher) → review-
(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #10)
> Comment on attachment 8900807 [details]
> Bug 1387377 - Don't run gtest or awsy on devedition builds;
> 
> https://reviewboard.mozilla.org/r/172252/#review177518
> 
> I am concerned that we are adding test sets and the more we have the greater
> the chance of making mistakes, this is fine if there is not another option,
> could we edit the tests.yml file and ensure gtest doesn't run on devedition
> (like it is for win10 right now):
> http://searchfox.org/mozilla-central/source/taskcluster/ci/test/tests.yml#226

Ah yes that's better, and I'll do that for awsy also, thanks!
Comment on attachment 8900807 [details]
Bug 1387377 - Don't run gtest or awsy on devedition builds;

https://reviewboard.mozilla.org/r/172252/#review177556

that looks good
Attachment #8900807 - Flags: review?(jmaher) → review+
Thanks, I'm trying to use ./mach taskgraph to verify the config is correct but I am still seeing gtest on devedition, not sure I am using it correctly and the help is a bit confusing.

:wcosta, how can I use ./mach taskgraph to verify that my config changes in tests.yml are correct?

I tried ./mach taskgraph target before and after my patch, but the output is the same. I also tried './mach taskgraph tasks -p pointing to the latest parameters.yml on central, running that before and after my patch applied locally and still the output is the same... I must be doing it incorrectly.
Flags: needinfo?(wcosta)
(In reply to Robert Wood [:rwood] from comment #15)
> Thanks, I'm trying to use ./mach taskgraph to verify the config is correct
> but I am still seeing gtest on devedition, not sure I am using it correctly
> and the help is a bit confusing.
> 
> :wcosta, how can I use ./mach taskgraph to verify that my config changes in
> tests.yml are correct?
> 
> I tried ./mach taskgraph target before and after my patch, but the output is
> the same. I also tried './mach taskgraph tasks -p pointing to the latest
> parameters.yml on central, running that before and after my patch applied
> locally and still the output is the same... I must be doing it incorrectly.

Hi, sorry for the delay. Usually, what I do is do a try push, grab the mach command line from decision task log and run it locally from now on.
Flags: needinfo?(wcosta)
(In reply to Wander Lairson Costa [:wcosta] from comment #16)
> 
> Hi, sorry for the delay. Usually, what I do is do a try push, grab the mach
> command line from decision task log and run it locally from now on.

Thanks :wcosta, I did that and I don't see any target tasks for 'awsy' or 'gtest' on devedition, so going to land this
Pushed by rwood@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/40bc1d4ce54b
Don't run gtest or awsy on devedition builds; r=jmaher
please request uplift to mozilla-beta here.
This is test-only, no need for an approval request.
Whiteboard: [PI:August] → [PI:August][checkin-needed-beta]
https://hg.mozilla.org/releases/mozilla-beta/rev/67363a144710
Whiteboard: [PI:August][checkin-needed-beta] → [PI:August]
https://hg.mozilla.org/mozilla-central/rev/40bc1d4ce54b
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla57
You need to log in before you can comment on or make changes to this bug.