913244 - Failed xperf tests should upload .etl file

Status: RESOLVED FIXED

(Testing :: Talos, defect)

Description

[(No longer employed by Mozilla) Aaron Klotz]

In cases such as bug 910759, we might see intermittent xperf failures that are triggered by run-time dynamic linking (including delay loading) or other variations in run-time behavior. Providing the .etl file would be beneficial because we could use that to perform further analysis.

Once 749421 lands we need to do two things:

1) Ensure that xperf is configured to collect stack traces (this will probably require us to fork off some additional bugs to set all of this up); and
2) Implement uploading of the .etl file that was generated by xperf.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

what do we need to do to collect stack traces?  we could start working on that now until the upload stuff is ready to use.

Comment 2

[(No longer employed by Mozilla) Aaron Klotz]

All 64-bit Windows machines that run xperf need to have the DisablePagingExecutive registry setting as outlined in https://developer.mozilla.org/en-US/docs/Performance/Profiling_with_Xperf. I'm not sure if this has already been done or not.

The other part is just a change in the command line that we use to invoke xperf.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

once we have DisablePagingExecutive done, we can adjust our xperf cli and then start the work on MOZ_UPLOAD_FILE from the harness.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

since DisablePagingExecutive is already being rolled out, lets figure out the xperf cli that we need.  Would it be -a stack:
http://msdn.microsoft.com/en-us/library/windows/desktop/hh162973.aspx

then we need to sort out the specific details and I could test them on try server.

Comment 5

[(No longer employed by Mozilla) Aaron Klotz]

Specific probes need to be selected for stack trace coverage. I'll make a patch with the changes.

Comment 6

[(No longer employed by Mozilla) Aaron Klotz]

Attachment: stackwalk config change

Looks like we already supported this, I just needed to make some additions to the list of probes that we want stacks for.

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 804626 [details] [diff] [review]
stackwalk config change

Review of attachment 804626 [details] [diff] [review]:
-----------------------------------------------------------------

umm, this was too easy.  Let me push this to try and see if it all works out well.

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

tested on try server, all is well:
https://hg.mozilla.org/build/talos/rev/ebb3e09205f5

next steps are to do the MOZ_UPLOAD_FILE work (which is only available on cedar right now).

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

We want to upload a test.etl file (which is a summary etl of the user and kernel traces captured).  it was mentioned that making a .zip file would compress this well.

As it stands, I am testing uploading a .etl file from Try server via the blobber toolchain.  Once that is finalized, I can post a patch to review and roll out.

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: upload xperf .etl files to blobber (1.0)

saw this working on try server.

Comment 11

[Jeff Hammel]

Comment on attachment 8333841 [details] [diff] [review]
upload xperf .etl files to blobber (1.0)

Review of attachment 8333841 [details] [diff] [review]:
-----------------------------------------------------------------

So this patch confuses me.  I don't understand how it is supposed to solve the bug.  So r-, although I could just be missing the obvious

::: talos/ttest.py
@@ +366,4 @@
>                      if test_config['setup']:
>                          # Generate bcontroller.yml for xperf
>                          utils.GenerateTalosConfig(command_args, browser_config, test_config)
> +                        setup = talosProcess.talosProcess(['python'] + test_config['setup'].split(), env=os.environ.copy())

This is already the default in http://hg.mozilla.org/build/talos/file/55fd35f46f08/talos/talosProcess.py#l18 (unless the environment is changing before this function call? if so, please add a comment).

@@ +390,4 @@
>                      if test_config['cleanup']:
>                          #HACK: add the pid to support xperf where we require the pid in post processing
>                          utils.GenerateTalosConfig(command_args, browser_config, test_config, pid=self.pid)
> +                        cleanup = talosProcess.talosProcess(['python'] + test_config['cleanup'].split(), env=os.environ.copy())

As above

::: talos/xtalos/etlparser.py
@@ +394,5 @@
>              errorFile.close()
>  
> +        mud = os.environ.get('MOZ_UPLOAD_DIR', None)
> +        if mud:
> +            mud_filename = os.path.join(mud, etl_filename)

So you set a variable and don't use it?  I don't understand

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: upload xperf .etl files to blobber (1.1)

forgot the os.rename

Comment 13

[Jeff Hammel]

Comment on attachment 8333955 [details] [diff] [review]
upload xperf .etl files to blobber (1.1)

Review of attachment 8333955 [details] [diff] [review]:
-----------------------------------------------------------------

> >                      if test_config['setup']:
> >                          # Generate bcontroller.yml for xperf
> >                          utils.GenerateTalosConfig(command_args,
> browser_config, test_config)
> > +                        setup =
> talosProcess.talosProcess(['python'] +
> test_config['setup'].split(), env=os.environ.copy())
> This is already the default in http://hg.mozilla.org/build/talos/fi
> le/55fd35f46f08/talos/talosProcess.py#l18 (unless the environment
> is changing before this function call? if so, please add a
> comment).

For the talosProcess instantiation, this still applies.

Comment 14

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: move .etl file to upload directory if we fail (2.0)

3rd times a charm?

Comment 15

[Jeff Hammel]

Comment on attachment 8333971 [details] [diff] [review]
move .etl file to upload directory if we fail (2.0)

Review of attachment 8333971 [details] [diff] [review]:
-----------------------------------------------------------------

looks great, thanks!

Comment 16

[Joel Maher ( :jmaher ) (UTC -8)]

http://hg.mozilla.org/build/talos/rev/635c822fd27e