1462007 - Use base profiles from testing/profiles in marionette tests

Status: NEW

(Testing :: Marionette Client and Harness, enhancement, P3)

Description

[Andrew Halberstadt [:ahal]]

Follow up work from bug 1451159.

This will move the marionette preferences out of testing/marionette/client/marionette_driver/geckoinstance.py and into the new testing/profiles system.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Is testing/profiles a different python package? Or how does it work? Which overhead would that cause for us if a new pref needs to be added or an existing one changed/deleted?

Comment 2

[Andrew Halberstadt [:ahal]]

No it's not a package. Most harnesses either find testing/profiles directly in the srcdir or we copy testing/profiles into the relevant tests.zip.

Given that marionette needs to be released to pypi, that means we either need to figure out how to get this directory included in the python package (maybe there's a way to include files outside of the marionette dir?), or we get marionette to download testing/profiles from hg.mozilla.org (this is what WPT does, though the infrastructure to do this existed before my changes and it's fairly complex).

If it ends up being too much trouble WONTFIX would be the third option. Ftr I filed this bug based on :ato and I's conversation on the newsgroup. I don't have too many strong opinions on the approach here or even whether it's worth doing.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Thanks for the additional information. So lets keep it open and see what could be done. I feel that packaging the preferences in Marionette client should be mandatory given that Marionette is kinda different from other in-tree harnesses.

Comment 4

[Andreas Tolfsen ❲:ato❳]

A packaging rule that includes the file at distribution time seems
like the way forward here.

Another complexity I fear it is harder to find a good solution to,
is that it the Marionette Python client is currently used out-of-tree
against all release channels of Firefox.  If the preferences found
under testing/profiles only tracks prefs that are currently used
in-tree in the tip-of-tree Firefox, then this will break
marionette_driver’s backwards compatibility.

The way we work around this at the moment for geckoinstance.py is
that individual preferences are annotated with a comment saying
when they can safely be removed.  This will typically take the form
of saying that a deprecated preferences can only be removed once
the Firefox version it unshipped/removed/renamed in makes it to
stable.

This system is imperfect for many reasons.  Firstly we don’t have
an established practice for maintaining preference backwards
compatibility in central, and we have seen multiple examples of
developers grepping for the preference they are removing and making
changes to geckoinstance.py without module peer approval or
understanding that it may break backwards compatibility for the
Python client.  Secondly I think it is fair to say it is a fragile
system that requires unreasonable amounts of manual labour to
maintain.

One could imagine an alternate approach where preference files
stored under testing/profiles are “versioned” to match the Firefox
version they shipped with.  When a new Firefox is released, we make
a copy of the user preference file and annotate it with the new
version number.  This would let us track preference changes back
in time, but would mean that the Python library would have to
synthesise a union of preferences across three different sets of
preferences.  I don’t know how this would work in practice, but it
seems theoretically possible.  Secondly, although I feel it may be
the best approach, it may require an insurmountable amount of
engineering unless we do it manually.

ahal: How do you think we should approach the backwards compatibility
problem?

whimboo: Can you confirm that marionette_driver is in fact being
used out-of-tree against earlier relased Firefoxen?

Comment 5

[Andrew Halberstadt [:ahal]]

Yeah, that's tricky, but I guess like you alluded to it's a problem that exists currently as well. I don't really have any bright ideas besides downloading the prefs from hg.m.o based on the version of the binary being used like WPT does.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Sorry, but I have no idea about out-of-tree usage of Marionette. Also on the new PyPI.org site I can no longer see download numbers. Not sure where to look for those.