Closed Bug 1507235 Opened Last year Closed 9 months ago

Add capability to run telemetry-tests-client locally

Categories

(Toolkit :: Telemetry, enhancement, P1)

enhancement

Tracking

()

RESOLVED FIXED
mozilla67
Tracking Status
firefox67 --- fixed

People

(Reporter: raphael, Assigned: grumpy)

Details

Attachments

(1 file, 2 obsolete files)

We need a way to run the telemetry-tests-client suite locally to efficiently develop tests for Firefox Client Telemetry.

Currently the following command fails on my setup on OS X with an ImportError:

./mach marionette-test \
    --gecko-log "-" \
    --binary /Applications/FirefoxNightly.app/Contents/MacOS/firefox-bin \
    --server-root=toolkit/components/telemetry/tests/marionette/harness/www \
    toolkit/components/telemetry/tests/marionette/tests/client/test_main_tab_scalars.py 

ImportError: No module named telemetry_harness.testcase

:davehunt and I developed a workaround, which can be used a starting point for this: https://gist.github.com/hackebrot/8303a0c75d897c840e30463527df52d4

With this patch applied, I am able to successfully run the telemetry-tests-client tests as you would expect.
Hey Raphael, are you going to work on this anytime soon? Do you have plans for that?
Flags: needinfo?(rpierzina)
Assignee: nobody → chartjes
Flags: needinfo?(rpierzina)
Hey Alessio, :chartjes and I discussed this last week!
(In reply to Raphael Pierzina [:raphael] UTC+01:00 from comment #2)
> Hey Alessio, :chartjes and I discussed this last week!

Cool, thanks! Marking this as P1 so that it shows in our team's dashboard and out of the triage list.
Priority: -- → P1
Getting the following error when trying to run the tests on my macos machine with the suggested patches applied

 ./mach marionette-test \
    --gecko-log "-" \
    --binary "/Applications/Firefox Nightly.app/Contents/MacOS/firefox-bin" \
    --server-root=toolkit/components/telemetry/tests/marionette/harness/www \
    toolkit/components/telemetry/tests/marionette/tests/client/test_main_tab_scalars.py
It looks like you tried to run a mach command from an invalid context. The marionette-test
command failed to meet the following conditions:

  is_firefox_or_android - Must have a Firefox or Android build.

Run |mach help| to show a list of all commands available to the current context.
Flags: needinfo?(rpierzina)
Please run `./mach bootstrap` and `./mach build` to set up a Desktop Artifact build on your machine and try again with the marionette-test command.
Flags: needinfo?(rpierzina)
No, we don't want to have those tests run via `mach marionette test`. Telemetry uses a custom harness, and as such would need it's own implementation of the mach command. See Firefox UI tests for example.

I have recently switched to using a Windows laptop and wanted to know what the steps are for getting the './mach bootstrap' and './mach build' working there.

I tried running them inside Powershell and get the following message

PS C:\Users\chartjes\mozilla-central> python mach build
Config object not found by mach.
0:03.03 Clobber not needed.
0:03.03 Adding make options from None
MOZ_OBJDIR=C:/Users/chartjes/mozilla-central/obj-
OBJDIR=C:/Users/chartjes/mozilla-central/obj-
Error running mach:

['build']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

Exception: Could not find a suitable make implementation.
Please use MozillaBuild 1.9 or newer

Flags: needinfo?(hskupin)

That's not related to this bug at all. Please ask in #build on IRC to get help.

Flags: needinfo?(hskupin)

A separate test runner is needed for Telemetry client tests

Pushed by rpierzina@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/875609b03809
Add capability to run telemetry-tests-client locally; r=raphael
Status: NEW → RESOLVED
Closed: 9 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla67
Attachment #9045024 - Attachment is obsolete: true
Attachment #9044951 - Attachment is obsolete: true
You need to log in before you can comment on or make changes to this bug.