1390084 - No gecko profile for Talos test with "mozharness: --geckoProfile" syntax

Status: RESOLVED FIXED

(Testing :: Talos, enhancement)

Description

[Shih-Chiang Chien [:schien] (UTC+8) (use ni? plz)]

"mozharness: --geckoProfile" syntax broke again on latest m-c. Expect to see corresponding artifact in Job Details tab.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=d871c2420a3db25e8fc9d27ed68a849479202582&selectedJob=122945869

Comment 1

[Shih-Chiang Chien [:schien] (UTC+8) (use ni? plz)]

This blocks my analysis and experiment for bug 1369632. Maybe @rwood can help?

Comment 2

[Robert Wood [:rwood]]

Hey :jmaher, I believe :ionut was looking at something similar in Bug 1354964, do you want him to take this one, or do you want me to have a look?

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

feel free to take a look, Ionut is on holiday all week long.

Comment 4

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=9353ca6568ccea9b20e449d9323ccf665d860be7

Comment 5

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=b596f44c6e45dff1928745c495e86a67a902143c

Comment 6

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=44f4c329bbfa9b73d79252c1d53f8f25a4e27dd4

Comment 7

[Robert Wood [:rwood]]

Hmm debugging on production, it seems the 'mozharness --geckoProfile' text is not making it's way through:

15:12:18     INFO - * RW * self.buildbot_config:
15:12:18     INFO - {u'properties': {u'buildnumber': 107, u'product': u'firefox', u'basedir': u'C:\\slave\\test', u'installer_path': u'public/build/target.zip', u'builddir': u'try_win10_64_test-tp6-e10s', u'repository': u'https://hg.mozilla.org/try', u'buildername': u'Windows 10 64-bit try talos tp6-e10s', u'stage_platform': u'win64', u'who': u'[e-mail]', u'project': u'', u'platform': u'win10_64', u'master': u'http://buildbot-master111.bb.releng.scl3.mozilla.com:8201/', u'slavebuilddir': u'test', u'taskId': u'SgaZR96eTUG3PxU4gGY9mA', u'branch': u'try', u'script_repo_revision': u'production', u'repo_path': u'try', u'slavename': u't-w1064-ix-039', u'revision': u'77e10f661fbd4711789452000860195e4b9e3c5e'}, u'sourcestamp': {u'repository': u'https://hg.mozilla.org/try', u'hasPatch': False, u'project': u'', u'branch': u'try', u'changes': [], u'revision': u'77e10f661fbd4711789452000860195e4b9e3c5e'}}

And results in catching this IndexError which just sets opts to None, and therefore not setting self.gecko_profile:

http://searchfox.org/mozilla-central/rev/48ea452803907f2575d81021e8678634e8067fc2/testing/mozharness/mozharness/mozilla/testing/talos.py#209

Comment 9

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=5b86a9cbe14267b7777718efe3ca4034060537e4

Comment 10

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e12770add5984001659eb246dfec859eda5aca7b

Comment 11

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=ae003d333b83de31f339f6817bfb3354fd46da7c

Comment 12

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=0608c5f182af3de46c689d53ace8ffaa35263551

Comment 13

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=9079c936d2907a4a8a575ef036307316371b3f31

Comment 14

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=df7388eb0eb1c802f74ba8a7adefd0f514076092

Comment 15

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e69b71354a6f172a5b26beb33c2d539ce6db777a

Comment 16

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=cef41c5a6b5e72c33baa17e5ee2904314f3dc449

Comment 17

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=60144e4a4d2626920a4ab4457214641c12d614f3

Comment 18

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=bffe901a979ddd9d54f8be76e3c59e303564cac3

Comment 19

[Robert Wood [:rwood]]

(In reply to Robert Wood [:rwood] from comment #18)
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=bffe901a979ddd9d54f8be76e3c59e303564cac3

Update: Patch makes gecko profiling work with it added in the try syntax, and the profiling data uploaded as an artifact; however now the "Performance" tab doesn't appear in treeherder when the job is selected.

Comment 20

[Robert Wood [:rwood]]

(In reply to Robert Wood [:rwood] from comment #19)
> (In reply to Robert Wood [:rwood] from comment #18)
> > https://treeherder.mozilla.org/#/jobs?repo=try&revision=bffe901a979ddd9d54f8be76e3c59e303564cac3
> 
> Update: Patch makes gecko profiling work with it added in the try syntax,
> and the profiling data uploaded as an artifact; however now the
> "Performance" tab doesn't appear in treeherder when the job is selected.

Hey :jmaher, so this makes sense as currently in Talos we aren't dumping out PERFHERDER_DATA when gecko profiling is turned on. However, in which case when a talos job with profiling is selected in treeherder, no "Performance" tab will appear, and therefore the profiling artifacts must be grabbed via the "Job Details" treeherder tab instead. I'm assuming that is acceptable? Or should I file another bug to change that?

Comment 21

[Robert Wood [:rwood]]

From the Description, "Expect to see corresponding artifact in Job Details tab." so I think that must be expected behaviour.

Comment 22

[Kris Maglione [:kmag]]

The performance numbers aren't particularly useful when the profiler is enabled, so I think that's fine. As long as we have performance results when the profiler is not enabled and profile results when it is :)

Comment 23

[Robert Wood [:rwood]]

Thanks :kmag, makes sense.

And this behaviour has been around for awhile, even when it was "spsProfiling":

http://searchfox.org/mozilla-central/rev/cd82cacec2cf734768827ff85ba2dba90a534c5e/testing/talos/talos/output.py#188

Comment 24

[Robert Wood [:rwood]]

Attachment: Bug 1390084 - Fix enabling gecko profiling on try;

Review commit: https://reviewboard.mozilla.org/r/173916/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/173916/

Comment 25

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8902352 [details]
Bug 1390084 - Fix enabling gecko profiling on try;

https://reviewboard.mozilla.org/r/173916/#review179266

thanks for figuring this out.  I agree that we don't want perf data in perfherder while using --geckoProfile, so glad that is working!

::: testing/mozharness/mozharness/mozilla/testing/talos.py:230
(Diff revision 1)
> -              except ValueError:
> +                except ValueError:
> -                  pass
> +                    pass
> +            else:
> +                # no opts, check for '--geckoProfile' in try message text directly
> +                if self.try_message_has_flag('geckoProfile'):
> +                    self.gecko_profile = True

in this scenarion what is self.gecko_profile_interval?  I assume it defaults to something useful?

Comment 26

[Robert Wood [:rwood]]

(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #25)
> Comment on attachment 8902352 [details]
> Bug 1390084 - Fix to turn on talos gecko profiling via try syntax;
> 
> https://reviewboard.mozilla.org/r/173916/#review179266
> 
> thanks for figuring this out.  I agree that we don't want perf data in
> perfherder while using --geckoProfile, so glad that is working!
> 
> ::: testing/mozharness/mozharness/mozilla/testing/talos.py:230
> (Diff revision 1)
> > -              except ValueError:
> > +                except ValueError:
> > -                  pass
> > +                    pass
> > +            else:
> > +                # no opts, check for '--geckoProfile' in try message text directly
> > +                if self.try_message_has_flag('geckoProfile'):
> > +                    self.gecko_profile = True
> 
> in this scenarion what is self.gecko_profile_interval?  I assume it defaults
> to something useful?

Thanks, yes gecko_profile_interval defaults to 1:

http://searchfox.org/mozilla-central/rev/51b3d67a5ec1758bd2fe7d7b6e75ad6b6b5da223/testing/talos/talos/config.py#32

Comment 27

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c98c89a11df2
Fix to turn on talos gecko profiling via try syntax; r=jmaher

Comment 28

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Backed out for mass environment setup failures:

https://hg.mozilla.org/integration/autoland/rev/6c192f0ce126e1386d4708267b1a54ccd3c5906c

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=c98c89a11df28e970c975088f1cfefc847aa06eb&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=usercancel&filter-resultStatus=runnable&filter-resultStatus=retry
Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=127067058&repo=autoland

[task 2017-08-30T14:34:20.741311Z] 14:34:20     INFO - retry: Calling rmtree with args: ('/builds/worker/workspace/build',), kwargs: {}, attempt #1
[task 2017-08-30T14:34:20.747083Z] 14:34:20     INFO - Running post-action listener: _resource_record_post_action
[task 2017-08-30T14:34:20.749013Z] 14:34:20     INFO - [mozharness: 2017-08-30 14:34:20.748730Z] Finished clobber step (success)
[task 2017-08-30T14:34:20.751838Z] 14:34:20     INFO - [mozharness: 2017-08-30 14:34:20.751552Z] Skipping read-buildbot-config step.
[task 2017-08-30T14:34:20.753480Z] 14:34:20     INFO - [mozharness: 2017-08-30 14:34:20.752207Z] Running download-and-extract step.
[task 2017-08-30T14:34:20.755107Z] 14:34:20     INFO - Running pre-action listener: _pre_download_and_extract
[task 2017-08-30T14:34:20.769009Z] 14:34:20    ERROR - Exception during pre-action for download-and-extract: Traceback (most recent call last):
[task 2017-08-30T14:34:20.775544Z] 14:34:20    ERROR -   File "/builds/worker/workspace/mozharness/mozharness/base/script.py", line 1982, in run_action
[task 2017-08-30T14:34:20.778866Z] 14:34:20    ERROR -     method(action)
[task 2017-08-30T14:34:20.782692Z] 14:34:20    ERROR -   File "/builds/worker/workspace/mozharness/scripts/desktop_unittest.py", line 533, in _pre_download_and_extract
[task 2017-08-30T14:34:20.785071Z] 14:34:20    ERROR -     if not self.try_message_has_flag('artifact'):
[task 2017-08-30T14:34:20.787197Z] 14:34:20    ERROR -   File "/builds/worker/workspace/mozharness/mozharness/mozilla/testing/try_tools.py", line 155, in try_message_has_flag
[task 2017-08-30T14:34:20.789555Z] 14:34:20    ERROR -     message = message or self._extract_try_message()
[task 2017-08-30T14:34:20.791579Z] 14:34:20    ERROR -   File "/builds/worker/workspace/mozharness/mozharness/mozilla/testing/try_tools.py", line 90, in _extract_try_message
[task 2017-08-30T14:34:20.793585Z] 14:34:20    ERROR -     elif self._is_try():
[task 2017-08-30T14:34:20.795751Z] 14:34:20    ERROR -   File "/builds/worker/workspace/mozharness/mozharness/mozilla/testing/try_tools.py", line 166, in _is_try
[task 2017-08-30T14:34:20.799745Z] 14:34:20    ERROR -     return ('try' in self.config.get('branch', repo_path) or
[task 2017-08-30T14:34:20.803643Z] 14:34:20    ERROR - TypeError: argument of type 'NoneType' is not iterable

Comment 29

[Robert Wood [:rwood]]

Thanks Sebastian, apologies, patch worked fine on try but obviously I missed something when it comes to the other branches.

Comment 30

[Robert Wood [:rwood]]

Comment on attachment 8902352 [details]
Bug 1390084 - Fix enabling gecko profiling on try;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/173916/diff/1-2/

Comment 31

[Robert Wood [:rwood]]

Updated patch on try, with and without profiling:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=60ff9fcbbeb2d496b161a3c375332abc949ffdd8

https://treeherder.mozilla.org/#/jobs?repo=try&revision=58e7e7520b14e5e3b82047e210567f076cc7f8b8

Comment 32

[Robert Wood [:rwood]]

Comment on attachment 8902352 [details]
Bug 1390084 - Fix enabling gecko profiling on try;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/173916/diff/2-3/

Comment 33

[Robert Wood [:rwood]]

Comment on attachment 8902352 [details]
Bug 1390084 - Fix enabling gecko profiling on try;

Not my day, updated again

Comment 34

[Robert Wood [:rwood]]

One more time...

https://treeherder.mozilla.org/#/jobs?repo=try&revision=4758c244276fdb4784c9990cfb68297564a4561c

https://treeherder.mozilla.org/#/jobs?repo=try&revision=314b195d00b74f4ce3c6a3c524bfb72e9bcc1da0

Comment 35

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8902352 [details]
Bug 1390084 - Fix enabling gecko profiling on try;

https://reviewboard.mozilla.org/r/173916/#review179682

sort of a rubber stamp here

Comment 36

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b4ea43ebf346
Fix enabling gecko profiling on try; r=jmaher

Comment 37

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/b4ea43ebf346

Comment 38

[Kris Maglione [:kmag]]

Thank you :)