1507235 - Add capability to run telemetry-tests-client locally

Status: RESOLVED FIXED

(Toolkit :: Telemetry, enhancement, P1)

Description

[Raphael Aurich [:raphael] UTC+01:00]

We need a way to run the telemetry-tests-client suite locally to efficiently develop tests for Firefox Client Telemetry.

Currently the following command fails on my setup on OS X with an ImportError:

./mach marionette-test \
    --gecko-log "-" \
    --binary /Applications/FirefoxNightly.app/Contents/MacOS/firefox-bin \
    --server-root=toolkit/components/telemetry/tests/marionette/harness/www \
    toolkit/components/telemetry/tests/marionette/tests/client/test_main_tab_scalars.py 

ImportError: No module named telemetry_harness.testcase

:davehunt and I developed a workaround, which can be used a starting point for this: https://gist.github.com/hackebrot/8303a0c75d897c840e30463527df52d4

With this patch applied, I am able to successfully run the telemetry-tests-client tests as you would expect.

Comment 1

[Alessio Placitelli [:Dexter]]

Hey Raphael, are you going to work on this anytime soon? Do you have plans for that?

Comment 2

[Raphael Aurich [:raphael] UTC+01:00]

Hey Alessio, :chartjes and I discussed this last week!

Comment 3

[Alessio Placitelli [:Dexter]]

(In reply to Raphael Pierzina [:raphael] UTC+01:00 from comment #2)
> Hey Alessio, :chartjes and I discussed this last week!

Cool, thanks! Marking this as P1 so that it shows in our team's dashboard and out of the triage list.

Comment 4

[Chris Hartjes [:grumpy][:chartjes]]

Getting the following error when trying to run the tests on my macos machine with the suggested patches applied

 ./mach marionette-test \
    --gecko-log "-" \
    --binary "/Applications/Firefox Nightly.app/Contents/MacOS/firefox-bin" \
    --server-root=toolkit/components/telemetry/tests/marionette/harness/www \
    toolkit/components/telemetry/tests/marionette/tests/client/test_main_tab_scalars.py
It looks like you tried to run a mach command from an invalid context. The marionette-test
command failed to meet the following conditions:

  is_firefox_or_android - Must have a Firefox or Android build.

Run |mach help| to show a list of all commands available to the current context.

Comment 5

[Raphael Aurich [:raphael] UTC+01:00]

Please run `./mach bootstrap` and `./mach build` to set up a Desktop Artifact build on your machine and try again with the marionette-test command.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

No, we don't want to have those tests run via `mach marionette test`. Telemetry uses a custom harness, and as such would need it's own implementation of the mach command. See Firefox UI tests for example.

Comment 7

[Raphael Aurich [:raphael] UTC+01:00]

https://searchfox.org/mozilla-central/source/testing/firefox-ui/mach_commands.py

Comment 8

[Chris Hartjes [:grumpy][:chartjes]]

I have recently switched to using a Windows laptop and wanted to know what the steps are for getting the './mach bootstrap' and './mach build' working there.

I tried running them inside Powershell and get the following message

PS C:\Users\chartjes\mozilla-central> python mach build
Config object not found by mach.
0:03.03 Clobber not needed.
0:03.03 Adding make options from None
MOZ_OBJDIR=C:/Users/chartjes/mozilla-central/obj-
OBJDIR=C:/Users/chartjes/mozilla-central/obj-
Error running mach:

['build']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

Exception: Could not find a suitable make implementation.
Please use MozillaBuild 1.9 or newer

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2]]

That's not related to this bug at all. Please ask in #build on IRC to get help.

Comment 10

[Chris Hartjes [:grumpy][:chartjes]]

Attachment: Add capability to run telemetry-tests-client locally

A separate test runner is needed for Telemetry client tests

Comment 11

[Chris Hartjes [:grumpy][:chartjes]]

Attachment: Bug 1507235 - Add capability to run telemetry-tests-client locally; r?raphael

Comment 12

[Chris Hartjes [:grumpy][:chartjes]]

Attachment: Bug 1507235 - Add capability to run telemetry-tests-client locally; r?raphael

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/875609b03809
Add capability to run telemetry-tests-client locally; r=raphael

Comment 14

[Oana Pop-Rus]

https://hg.mozilla.org/mozilla-central/rev/875609b03809