Open Bug 1462007 Opened 2 years ago Updated 1 year ago
Use base profiles from testing/profiles in marionette tests
Follow up work from bug 1451159. This will move the marionette preferences out of testing/marionette/client/marionette_driver/geckoinstance.py and into the new testing/profiles system.
Summary: Use base profile s from testing/profiles in marionette tests → Use base profiles from testing/profiles in marionette tests
Is testing/profiles a different python package? Or how does it work? Which overhead would that cause for us if a new pref needs to be added or an existing one changed/deleted?
No it's not a package. Most harnesses either find testing/profiles directly in the srcdir or we copy testing/profiles into the relevant tests.zip. Given that marionette needs to be released to pypi, that means we either need to figure out how to get this directory included in the python package (maybe there's a way to include files outside of the marionette dir?), or we get marionette to download testing/profiles from hg.mozilla.org (this is what WPT does, though the infrastructure to do this existed before my changes and it's fairly complex). If it ends up being too much trouble WONTFIX would be the third option. Ftr I filed this bug based on :ato and I's conversation on the newsgroup. I don't have too many strong opinions on the approach here or even whether it's worth doing.
Thanks for the additional information. So lets keep it open and see what could be done. I feel that packaging the preferences in Marionette client should be mandatory given that Marionette is kinda different from other in-tree harnesses.
A packaging rule that includes the file at distribution time seems like the way forward here. Another complexity I fear it is harder to find a good solution to, is that it the Marionette Python client is currently used out-of-tree against all release channels of Firefox. If the preferences found under testing/profiles only tracks prefs that are currently used in-tree in the tip-of-tree Firefox, then this will break marionette_driver’s backwards compatibility. The way we work around this at the moment for geckoinstance.py is that individual preferences are annotated with a comment saying when they can safely be removed. This will typically take the form of saying that a deprecated preferences can only be removed once the Firefox version it unshipped/removed/renamed in makes it to stable. This system is imperfect for many reasons. Firstly we don’t have an established practice for maintaining preference backwards compatibility in central, and we have seen multiple examples of developers grepping for the preference they are removing and making changes to geckoinstance.py without module peer approval or understanding that it may break backwards compatibility for the Python client. Secondly I think it is fair to say it is a fragile system that requires unreasonable amounts of manual labour to maintain. One could imagine an alternate approach where preference files stored under testing/profiles are “versioned” to match the Firefox version they shipped with. When a new Firefox is released, we make a copy of the user preference file and annotate it with the new version number. This would let us track preference changes back in time, but would mean that the Python library would have to synthesise a union of preferences across three different sets of preferences. I don’t know how this would work in practice, but it seems theoretically possible. Secondly, although I feel it may be the best approach, it may require an insurmountable amount of engineering unless we do it manually. ahal: How do you think we should approach the backwards compatibility problem? whimboo: Can you confirm that marionette_driver is in fact being used out-of-tree against earlier relased Firefoxen?
Yeah, that's tricky, but I guess like you alluded to it's a problem that exists currently as well. I don't really have any bright ideas besides downloading the prefs from hg.m.o based on the version of the binary being used like WPT does.
Sorry, but I have no idea about out-of-tree usage of Marionette. Also on the new PyPI.org site I can no longer see download numbers. Not sure where to look for those.
Depends on: 1553949
You need to log in before you can comment on or make changes to this bug.